261. Social Justice Assessments

Traditional methods of assessing student learning favor those students that reside in well-resourced school districts while leaving low-income students at a substantial disadvantage. These grading systems also encourage students to focus on their grades rather than on their learning. In this episode, Judith Littlejohn, Meghanne Freivald, and Katelyn Prager join us to discuss a variety of social justice assessment techniques that can help to create a more equitable environment in which all students can be successful.

Judie is the Director of Online Learning at SUNY Genesee Community College, Meghanne is an Instructional Technology Specialist at Alfred University, and Katelyn is an Assistant Professor in the English Department at the Fashion Institute of Technology.  Judie, Meghan, and Katelyn worked together on a SUNY Faculty Advisory Council on Teaching and Technology committee on social justice assessments.

Show Notes

Transcript

John: Traditional methods of assessing student learning favor those students that reside in well-resourced school districts while leaving low-income students at a substantial disadvantage. These grading systems also encourage students to focus on their grades rather than on their learning. In this episode, we explore a variety of social justice assessment techniques that can create a more equitable environment in which all students can be successful.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guests today are Judith Littlejohn, Meghanne Freivald, and Katelyn Prager. Judie is the Director of Online Learning at SUNY Genesee Community College, Meghanne is an Instructional Technology Specialist at Alfred University, and Katelyn is an Assistant Professor in the English Department at the Fashion Institute of Technology. Judie, Meghan, and Katelyn worked together on a SUNY Faculty Advisory Council on Teaching and Technology committee on social justice assessments. Welcome Meghanne and Katelyn and welcome back, Judie.

Meghanne: Thank you.

Katelyn: Thank you.

Judie: Thank you.

John: Today’s teas are:

Judie: …I have Lady Grey.

Rebecca: That’s a good one…

Judie: …In my DTL mug.

John: …a nice Desire to Learn mug.

Meghanne: I have iced green.

Rebecca: And Katelyn, how about you?

Katelyn: Mine’s water right now, if it were the evening, I would have one bag of peppermint and one bag of chamomile together, delicious.

Rebecca: Sounds nice and calming.

Rebecca: I have hot cinnamon spice tea.

John: And I have black raspberry green tea.

Rebecca: We’ve invited you here today to discuss your work on social justice assessment. Perhaps, we can start with a discussion on what you mean by social justice assessment.

Judie: So social justice assessment considers factors such as race, culture, language proficiency, socioeconomic status, and ability while working to dismantle systems of power, bias, and oppression in evaluation of student learning. So various approaches including equitable assessment, labor based grading, and ungrading, as they relate to the purpose, process, wording, and structure of student learning assessments are included. So we’re trying to focus on the learning that our diverse students achieve as it relates to specific learning outcomes just to mitigate the influence of dominant norms on our students’ grades. So we’ve all been working together for the last couple of years on a SUNY task group that was part of the Faculty Advisory Council on Teaching and Technology, which I chair. So we’re a subcommittee of an Innovations in Assessment group, and there’s a couple more of us who couldn’t make it today, but we’ve been a really close-knit group, I think, working together for over two years. And we really enjoyed the project, which resulted in a website with all these artifacts on it that people will be able to access. And we’re hoping down the road that we can continue our work, but we’ll get to that later on in this conversation.

John: And we’ll share a link to the overall website as well as your group-specific component of that in the show notes. So this was partly implied in your response defining social justice assessment, but, what are some of the shortcomings of traditional grading systems in terms of equity?

Meghanne: When we were doing our research on this topic, we encountered many drawbacks of the traditional types of assessments that we all experienced all the way up through school and into college, and I’ll share a few of them. One is that the focus is often on the grade rather than the actual learning process and what the student will actually be able to do, and be able to learn as a result of engaging in the education process. They just focus on the grade, “what’s my grade?” and that sort of misses the point. It creates a system where students are compared to each other rather than having the focus be on individual growth and achievement. It also can put students at an advantage or disadvantage based on characteristics such as race, ethnicity, socioeconomic status, disability status, language proficiency, and lots of other characteristics that students themselves don’t have any control over. We found in our research that traditional assessments tend to favor white, affluent, high-achieving students, and that really isn’t who most of our students are anymore. So we really need to remove barriers and create a way for students to accurately represent the learning that has taken place.

Rebecca: So you hinted to this in your response about traditional grading systems comparing students to one another. So thinking about that, what role should students play in determining how their learning is assessed?

Katelyn: I’ll tackle that one, and I want to answer it with a disclaimer to start because social justice assessment is an umbrella term that has all of these different strategies that are wrapped up in it, and each of those approaches, whether it’s ungrading, or labor-based grading, might have a slightly different response to that question. They all share the same goal, that students should not be systematically disadvantaged by the assessment mechanisms, and that we want to increase student agency in the classroom. We want students to be active participants in their own learning, but the actual question of how students might participate in determining their own assessments might look very different depending on who you’re talking to and what approach they use. Maybe it’s literally helping design the assessment mechanisms, the grading contract, grading rubric, maybe it’s creating flexible assignments that allow students to determine what learning is being assessed, or in the case of ungrading, maybe it’s just deprioritizing the assessment entirely in order to emphasize the individual student’s learning journey through the course. So I guess my answer, tentatively to your question, is yes, students should be participants in determining how they’re learning is assessed by the big how, and why is going to differ.

John: As you noted, there’s a wide continuum of alternative grading policies that can fit under this category of social justice assessment. Some of them are not that much different than traditional practices, and others are quite a bit different. One approach, which is much closer to the traditional grading systems that people are already using is a system of mastery learning. Could you talk a little bit about what mastery learning is and how that could be used in the classroom to provide a bit more equity.

Judie: So mastery learning is, instead of assessing a student or evaluating a student with one assessment, and giving them that grade, the students are able to go back and revisit the content and work again on any material that they didn’t understand or try things over again. So it’s an iterative process, and they should get some sort of formative feedback in between attempts so that they can understand what it is they need to work on and focus on. And this way, it’s more equitable, because the students are able to take the amount of time that they need to work on the assessment, they can access any review materials that they need to establish their foundational knowledge and continue on. And it just really helps the students learn and grow. And I think it’s a great way to establish foundational knowledge. I use it myself in all the history courses that I teach, and I just think it’s a great process. If you think about it, any athlete, that’s what they do. So if you’re learning how to play baseball, how many hours are spent in a batting cage, or like on the pitcher’s mound, how many times do you try again, and again, and then again, until you are able to do it correctly, or do things accurately? So I always liken it to use that sports analogy, because I really think that helps people understand that students’ learning… you have to practice and you can’t tell somebody something once and expect them to integrate it into all the knowledge they already have, and be able to recall it instantly. So I just think it’s a great way to level the playing field of students so that when you move on to the next part of your content, they all have the same foundation, and they’re ready to go forward.

John: And by explaining it to the students that way, in terms of a sports metaphor, it’s something that they can pretty easily connect to, and I think it also would help to promote a growth mindset, which we know is effective in increasing learning as well.

Rebecca: Another assessment strategy one might use is minimal or light grading that falls under this social justice umbrella, and is a bit different than mastery learning. Can you describe what minimal or light grading is?

Meghanne: Yeah, I’ve seen this described in a couple of different ways. This isn’t something that we really included in a lot of our research, so I kind of looked this up just a little while ago and it’s very interesting. And one approach is more on like the whole course level. And there’s another approach that can be taken on an assignment level. So for an entire course, what an instructor may do is that they would assign assignments throughout the semester, but most of them would not be graded, they would be used as like a conversation piece. And they would be discussed and gone over during class, which would then provide opportunities for the students to seek clarification and for the instructor to provide feedback in the moment. So then the assessment then becomes part of the learning process. So then when there are a small number of assessments that are given for a grade, then when the students get to those assessments, they’re not as intimidating. They’re things that they’ve done with their classmates, they’ve done them with their instructors, they’ve done them in class. So I think it’s a very interesting strategy because it removes a lot of the anxiety that students may have around assessment, because it’s just something that they’ve done in their class. Another take on this that I’ve seen is, on an assignment level, something like a paper, something that may require a lot of revision, where when the professor is grading that assessment, they would maybe not take the time to go through and mark all of the grammar and spelling and mechanical errors, but maybe they would look at a section of that, maybe point out some things the students are doing over and over again, but not mark up the entire paper, but just say, “Okay, these are the things you need to pay attention to that are recurring through your paper.” And then as they read and grade that student’s paper, they focus more on the message that the student is trying to convey and the ideas that they’re sharing, rather than the mechanics and the grammar and the spelling.

John: And one common thing I think, to both mastery learning and minimal light grading is that the goal is to provide students with feedback. In some cases that can be automated. Mastery learning systems involve some degree of automation, sometimes by textbook providers, or perhaps adaptive learning systems, or it could be questions that you put together. But if you’re going to provide feedback on writing, it can require a lot more time. And a minimal light grading approach allows faculty to provide feedback on the most important things without taking up as much time to allow faculty to provide feedback on a wider range of topics, which, again, is I think, to some extent in the same sort of spirit.

Rebecca: Light grading can help not intimidate a student with too much feedback. If you see just a paper completely marked up, it might feel like there’s no possibility for moving forward or revising. But emphasizing what’s most important to change, or most important to focus on can help a student prioritize. And this can be really important to someone new to a discipline who might not know what’s most important.

Katelyn: I’m so glad you said that.

Meghanne: There’s an element of trust there as well, because if we point out what a student needs to focus on mechanically or grammar wise in a small part of that paper, then they can be trusted to then use their judgment to go through it and read it more carefully, and then make those edits based on the feedback that they had received. So it is visually much less intimidating. Plus, it might be a motivating factor for some students too that their professor is trusting them to be in charge of that revision.

John: Another type of social justice assessment involves contract grading. Could one of you talk a little bit about how contract grading fits into this category of social justice assessment?

Katelyn: Sure, I think contract grading is one of those terms that’s gaining some broader popularity and recognition. So it’s probably a term that may be pretty familiar to a lot of instructors at this point. So maybe it doesn’t need a lot of explanation. I’ll just say there’s a couple of different models of contract grading. In some cases, the instructor might provide that contract at the start of the term. In other cases, the instructor and students would be able to negotiate that contract collaboratively together at the start of the term so that students have more of that active stake in the contract itself. Generally, the grading contract would lay out certain requirements which students would need to fulfill to receive their desired grade. And that might include requirements related to attending class or conferences, completing low-stakes assignments, completing major assignments, maybe some page- or process-based requirements. But the bottom line is that the contract gives students a clear picture from day one of the work required by the class so students can look at that contract and know exactly how much work they’re going to need to complete from day one, to get the grade that they really want to receive in the course. I think the additional benefit of contract grading for our conversation is that it decouples grades from assessment so students have more space to take risks in their work rather than aiming for correctness. And on the faculty side, faculty can respond to the content and spirit of the students work as opposed to justifying a grade. I think most important, though, because this system privileges students who are investing the time and effort into their learning, all students have the same potential to earn a high grade in the course regardless of their knowledge or ability with the subject matter prior to the start of the course. So to use another sports metaphor, it works to level the playing field on day one for students who may have very different levels of preparedness and experience with the subject matter.

Rebecca: Another strategy that folks might use, which we’ve certainly talked about quite a bit on this podcast at various times is peer assessment. Can you talk a little bit about what that looks like and how that fits into this social justice model?

Judie: So peer assessment, or I tend to call it peer review, helps to build student investment in writing, and helps the students understand the relationship between their writing and their coursework by helping them engage with the writing in a way that encourages more self reflection and works to help them build their critical thinking skills about their own work. And I think it also helps the students learn from one another, because they’re sort of trying to evaluate their peers’ work against the requirements for the course. But then you also look at your own writing in a new perspective, and you learn from what you’re seeing your peers write and from the feedback that you’re receiving from your peers.

John: Might students perhaps take feedback from their fellow classmates a bit more seriously than they do feedback from their instructors.

Judie: A lot of students self-report that they learn more from this peer review activity, because they’re trying to identify and articulate weaknesses that they’re seeing in their peers’ papers, and also in their own. And I think trying to incorporate feedback from both their peers and their instructor into their own work, I think, just helps raise that awareness and any kind of feedback that’s constructive, as they think about it and reiterate it and rewrite their work. It just helps with their critical thinking. And I think just raise awareness of how they write, and maybe they can be more thoughtful about what they’re writing going forward. I think they also, if they question their peers, say “How did you come up with this?I love this idea,” then they can apply some of this, that they’re learning from their peers to their own work, too. So perhaps that’s what you were getting at John, when you asked that question was, they may benefit more from their classmates telling them how they came up with their ideas than from their instructor just dictating what the expectations are.

Rebecca: I would expand the model to include not just writing but also other creative projects and things. It’s certainly a practice that’s pretty common in the arts, for example, to do peer review of student work.

John: And they also get to see what their peers are doing, which can serve as a positive role model. When students see that other people are doing something that they hadn’t considered doing, it could serve as a way of improving their work.

Katelyn: I think a lot of students come into the classroom thinking of their teacher as the sole reader or audience for their creations throughout the course of this semester. So anytime we can expand those audiences and have students thinking rhetorically about who else might be the consumer of their work. I think that that can benefit our students in really important ways.

Rebecca: It also seems like it’s a good opportunity to formulate community around an activity like that.

Katelyn: Absolutely.

John: One of the other areas you address with this group was the topic of labor-based grading, could you talk a little bit about that?

Meghanne: Yeah, labor-based grading removes the focus from the end product assignment and shifts it to the process of creating that piece of work. So students are provided with feedback throughout the process regarding their labor or the work that they put in. And they’re given opportunities to continue working to improve what they’re producing, and to achieve a desired grade based on a contract sometimes, so there is some overlap with contract grading, but not always. There typically aren’t penalties for students who revise and update their work, because that’s part of the learning process. And it really helps students determine what their end grade may be and how much effort they want to put in, because often, they will be given some sort of guideline for what different grades may be achieved based on certain levels of effort, or certain levels of work that are completed. And also there may be opportunities to grade based on completion rather than more of a subjective sort of qualitative grade.

John: So do you mean like using a light grading or minimal grading where you either completed satisfactorily or you haven’t, and as long as you complete a certain number of assignments or activities, you achieve that grade,

Meghanne: That or also if there’s criteria, like a rubric, and they hit all of the criteria, then they receive full credit.

John: Which becomes, actually, I think, a form of specifications grading.

Rebecca: And then one other model that you’ve talked a little bit about already today is ungrading. Can you expand upon that a little bit more?

Katelyn: Yeah, so ungrading works to deprioritize numerical grades or even attempt to eliminate them entirely. So I hope I’m not speaking out of turn when I say, I think that this is the most controversial of the approaches that we have been researching, it tends to get the most pushback from faculty because it is so different from what we have often been taught or trained to do. So instead of focusing on those numerical grades, instructors are encouraged to focus on providing learner feedback that encourages growth. Okay, I have a quote that is from an ungrading expert I’d like to share. This from Sean Michael Morris and he says, quote, “at the foundation of ungrading, lies something that could change school entirely. A suggestion that ranking and evaluation and the concomitant expertise of the ranker or evaluator is entirely an optional way of viewing things.” And I’m going to end the quote there because I think that that important kernel is that ungrading works to dismantle the hierarchy of the classroom and refocus the attention on individual student learning is an approach that requires a lot of trust between student and instructor, and a lot of student buy-in as well. Students have to be invested in the learning that’s going to happen throughout the course itself. And in a completely ungraded classroom, student grades might be based simply on a final student reflection, or even a one-on-one conversation between teacher and student about the grade that the student has earned. But because ungrading really rejects transactional grading systems, the final grade is more of an afterthought than an important outcome of the course, much less important than learning that’s occurred throughout the semester.

Rebecca: So today, we’re recording on August 9, James Lang posted on Twitter about how deep the system of creating actually is that there’s even things like discounts for insurance, for good students, or good grades. And that it’s really challenging to overcome a system that’s so ingrained beyond just our education system, but into many other systems as well. So I think that that, in part, is why there’s such a strong pushback on this particular method.

John: And we’ve always done it that way, at least for the last century or so.

Rebecca: Change is hard.

Katelyn: Yeah, I think that the traditional grading system is really embedded into not only academia but outside of academia as well. And even within a class that takes an ungrading approach, we still face that question at the end of the semester of “Well, what’s the grade going to be in the system?” because we don’t really have the option, at least at most institutions, to say, “No grade, job well done.” At least at my institution, I still have to put in a letter grade for the student. So we can work to reject that system as much as we can. But at the end of the day, we’re still operating within that same structure. And maybe that’s a question of what’s next, right? Like, are we going to see one day a future where more universities embrace this idea of learning for the sake of learning as opposed to learning for the grade? I don’t know.

John: One of the other things you address on the website is how perhaps the use of authentic assessment or UDL types of assessments might improve equity by providing a more equal playing field for students. Could you talk a little bit about how going beyond the traditional term papers and tests might provide a more equitable way of assessing students’ learning.

Judie: I think anytime you use authentic assessment that helps, or generally it allows the students the opportunity to demonstrate their learning in the way that works best for them. The students are writing a term paper, for example, they can write the paper the traditional way, or they can give a presentation or record a presentation, and still provide their citations and so forth at the end. Or they can do something visual, some sort of a PowerPoint or a nice visual display of the topic and again, cite their sources and explain their images to the group so that people understand how they’re meeting the learning outcome. And I feel like that’s just a good way if people are struggling with language, if people are just struggling with writing in general, I think that this levels the playing field, because it gives everybody an opportunity to really show their knowledge and shine and not just pigeonhole themselves into one more paper or one more multiple choice test, if they have test anxiety. Some of our traditional forms of testing or final assessment just set students up to fail. And allowing students to choose to demonstrate their learning in a way that they’re good at sets them up to succeed. And I think that’s what we really want at the end of the day. And of course UDL principles, those are Universal Design for Learning, and that does include equity in its heart. So that would definitely help to keep things equitable in the classroom. If you’re following UDL.

Rebecca: The multiple forms you were just talking about is a great example. [LAUGHTER]

Judie: Last semester, I had a student who, they’re supposed to do a blog post, and the student instead of writing a blog post, he made a video and he did it three different times. So one is on World War One, one’s on World War Two, and the third one was on revolutions, and so, this student stood in front of a whiteboard, and he had his camera set up so he could film himself. And he had his iPad in his hand. So he talked about a battle, say, for example, and he would draw it out on the board. And then he would show his citation on his iPad. And then he had other citations typed up and taped to the whiteboard. And he went on for 15 minutes, and just was making sure he explained things again, and drew little examples. And he was so animated, and so excited about his topic. And you’re not going to capture that on a written exam, or even in somebody’s written paper. It was just tremendous the way he was able to show all that he had learned and all that he was interested in, and the extra research that he had done, because he felt the freedom to pursue this topic, because he knew he was able to express it the way that suited him the past. And it was just amazing. So I think anytime we can incorporate these things, and I understand that there are times when, according to your creditor, or people have to sometimes sit for a specific certification, it doesn’t always fit, but I think if you can fit this type of assessment in, it is definitely worth it. Because just to see the joy in students when they can explore and expand their knowledge, and then feel confident in demonstrating that to you, it’s just tremendous.

Rebecca: I love the flexibility in demonstrating knowledge and understanding and skill sets because in some of our traditional methods, we are arbitrarily assessing something else. So we may be arbitrarily testing how well you can take a multiple choice test or how well you can take a test within a certain timeframe, or how well you can write, whether or not that’s actually the topic. So if I’m learning about history, there’s some learning objectives I’m trying to meet related to history that may or may not include writing. And if writing is not one of those outcomes that we’re hoping for, then we don’t need to be assessing it.

Judie: Exactly. He did this thing on medical advancements in World War One, it was just tremendous and he was so charming, because he just was so wrapped up in it that you just had to root for the guy. It was good.

Rebecca: I love that. So for those of us who may want to move towards equitable grading systems, what are some initial steps we might take? Because it could feel really daunting if you haven’t ventured down this path before.

Meghanne: Yeah, if you are not interested in overhauling your entire grading system, just to try this out, a nd to make your assessments more socially just, there are some adjustments that can be made to existing assignments. And really, the important thing is to consider the learning objectives and really think about what needs to be graded. So one of the things that we’ve talked about a lot in all of our different presentations that we’ve done is whether or not to grade for things like grammar and spelling, and mechanics, and English language proficiency. So in an example, like a discussion board, when you’re really interested in what the students have to say, and their interaction with each other, and the questions that they asked, does it really matter if their grammar and spelling is perfect in that instance, if they’re having a great conversation on a topic, and they’re learning from each other. So that’s one thing that we could suggest. Another is thinking about just the fact that sometimes students have challenges in their lives. They’re human beings, they have families, they have jobs, many of our students are athletes, and they have to travel and they have games and something like flexible due dates is very, very helpful for students because then they’re able to complete their work, certainly within a reasonable timeframe. But if those dates are a little bit more flexible, and they have access to those assignments in the learning management system beyond the actual due date, for instance, then that gives them the ability to complete that work without being penalized. So another mechanism would be in the learning management system, when students are taking quizzes, would be allowing backtracking, allowing students to go back and check their answers, that sometimes is a setting that a lot of professors really rely on, to try to avoid cheating. And as an LMS administrator, that is something that I see a lot. And I think that that can really be harmful to students, because many of our students are told to always go back and check your work. And if they’re not allowed to go back and check their work, that can be very frustrating. And also forcing completion is something that I would recommend turning off because again, that can create test anxiety. And often I think when completion is forced, there’s also a timer. So I think if any timers can be removed as well, then that does a couple of things. It can help remove testing anxiety. But then also, if there are students who require extra time due to a disability accommodation, then the professor at that point doesn’t have to go in and adjust all of the LMS settings for those students, because it’s already open ended and everyone can have as much time as they need to complete that assessment. So it really is just important to look at what the learning objectives are and what actually needs to be assessed. And the goal is always to remove barriers. So another thing that can be done is to just ask students, have a conversation about it, and find out what barriers they’ve experienced.

John: At the start of this. You mentioned the website that you were creating, could you talk a little bit more about what resources are there and how that might evolve over time?

Katelyn: Yeah, so the website, we have been slowly adding resources to over the past two years. And at this point, it’s becoming a pretty robust little outlet for people interested in social justice assessment. So, you go to the website, you can find an overview of the big picture theory of social justice assessment, as well as the various approaches that we’ve discussed today. We also have a really pretty large bibliography of resources for further reading for people who want to learn more about any one of these topics. And we’ve been working to develop a collection of sample assignments from faculty across SUNY. So we’re still working to collect additional sample assignments from faculty who might already be implementing some of these strategies within their classrooms. I think the more we can share those assignments with one another, the better off we’ll all be. I think a lot of us are doing social justice assessment in small ways in our classroom without realizing it. So the more we can share those resources and that knowledge, the more hopefully we can get people on board. So, hopefully, we’ll be able to share that link in the show notes. And people will be able to check that out.

Rebecca: So we always wrap up by asking: “What’s next?”

Judie: So for our little group, one thing that I think might be next for us is SUNY is updating the SUNY general education requirements that are mandated with the completion of any SUNY degree. And they’ve added a requirement for equity, inclusion, and diversity. So I’m hoping that our group can help contribute resources to that effort, and our website could be one more place where people go to for information on social justice assessment so that they can incorporate those into their courses that are designed to meet the DEI requirement.

Katelyn: Well, I’m gonna go take my one-year old to the pool. [LAUGHTER]

Judie: Nice.

Katelyn: I think, big picture, though, the “what’s next” I want to just give is, I hope that we’ll start to see more institutional support for some of these approaches. I think that there are still a lot of barriers, particularly for contingent faculty who want to embrace some of these practices. So I hope what’s next will be more departmental institutional support for this: more time, more resources, etc. But yeah, my personal what’s next is I’m gonna go enjoy this beautiful day.

Rebecca: Meghanne, do you want to add anything?

Meghanne: Sure yeah, at my institution, I am sharing this information, pretty much any chance I get, I’m meeting with our new incoming faculty in a couple of weeks. And this will be one of the topics that we discuss. And I’m also co-chair of our universal design for learning task force. And we have a few events and projects that we’re working on to spread the word on UDL, and also innovative assessments and social justice assessments as well.

Rebecca: Lots of great things coming and some really wonderful resources that you’ve shared today. Thank you so much for joining us.

Katelyn: Thank you.

Judie: Thank you for having us.

Meghanne: Yeah, thank you.

John: And thank you for all the great work you’ve done on this over the last couple of years and the resources you’re sharing.

Judie: I would just like to say that Shena Salvato is also in our group. She’s at Cortland, I believe. And Chris Price from SUNY is in our group, and they are missed today. They’ve been with us for all our other presentations. I know that Shana in particular wants to get the band back together and have some more meetings going forward so we can keep working together. And it was really good to see you guys again.

Katelyn: Likewise.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

257. PsycLearn

Adaptive learning platforms provide each student with a customized learning path based on the student’s individual learning needs. In this episode, Anna Yocom, Linda Goldberg, and Alan Strathman join us to discuss how the American Psychological Association has developed adaptive learning packages for core psychology courses.

Show Notes

Transcript

John: Adaptive learning platforms provide each student with a customized learning path based on the student’s individual learning needs. In this episode, we examine how one professional association has developed adaptive learning packages for core courses in their discipline.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are all involved with a psych learning project, which we’ll be talking about throughout the podcast. Let’s have each guest introduce themselves.

Anna: Hi, I am Anna Yocom. I am a senior lecturer at The Ohio State University in psychology and I am also a Content Manager for PsycLearn.

Linda: And I’m Linda Goldberg. And my role with the PsycLearn project is that of product evangelist.

Alan: Hi, this is Alan Strathman. I’m a content manager for PsycLearn and was a professor of psychology for a long time at the University of Missouri.

John: Welcome, everyone. We’re really happy to talk to you about PsycLearn. I first heard about this when I was working on a project at SUNY dealing with adaptive learning solutions. And we had seen some write ups of PsycLearn so we want to find out a little bit more.

Rebecca: Today’s teas are:… Alan, are you drinking tea?

Alan: I’m drinking coffee.

Rebecca: Oh, a coffee drinker.

Alan: Sorry.

Rebecca: I see how it is. [LAUGHTER]

Alan: I had tea this morning this morning.

Rebecca: Oh, that’s good. That’s good.

Rebecca:: How about you, Linda?

Linda: I’m having an iced tea this afternoon. It’s hot and muggy outside, so something citrusy and decaf.

Rebecca: …and perfectly refreshing. [LAUGHTER]

Linda: That’s right. And sweetened.

Rebecca: Awesome. Anna?

Anna: I had tea earlier, but I have switched to water.

John: And I’m drinking spring cherry green tea.

Rebecca: Nice. I have some Scottish breakfast. Still in my pot today.

John: We’ve invited all of you here today to discuss the PsycLearn adaptive learning platform developed by the American Psychological Association. Could you tell us a little bit about the origin of this project?

Linda: Okay, I’m going to start with that. And there are a couple of overlapping facets to the origin story, so to speak. One is really a desire to expand APA’s publishing program. APA has a very rich publishing program, which includes publishing of over 90 academic journals, a very robust books publishing program, where many of those books are aimed at clinical practice, as well as books that might support the professional researcher, books that might support self help, and so on. And we found that there was an opening for us to make a contribution to undergraduate academic instruction and contribute to that portion of the path toward the profession. We also were recognizing as we contemplated this new project that there was a tremendous amount of disruption in the higher educational publishing landscape, a lot of cost pushback, a lot of desire among publishers to leverage technology. And we thought that perhaps not simply for delivery of content, but to improve the learning experience and to improve efficacy that we saw there might be some opportunity there.

Alan: Yeah, also, let me jump in. I think the developers also saw this as an opportunity to work toward some of the goals that APA has for the field of psychology, including, for example, utilizing psychology to make an impact on social issues.

Linda: And the field of cognitive psychology is one that tells us a lot about how we learn. So we felt like we could draw from our very own field in development of our final product,

John: Which courses have been developed for this platform.

Linda: So we started with a course for research methods and psychology and added a PsycLearn statistics for the behavioral sciences to that. These two courses are typically required for all majors in the discipline. And they also represent the most challenging content for many students. So we thought that seemed like a good place to start.

Rebecca: Can you talk a little bit about how this material was developed?

Alan: Sure, we use a backwards design process where we start by identifying what we want students to learn in each module and a module is similar to what would be a chapter in a regular textbook. These outcomes that we have in mind become Learning Objectives, and they really guide the content development. Learning Objectives are developed through collaboration, a conversation, maybe a hashing out between the subject matter experts and the content managers, and are influenced by a variety of factors, including how well the learning objectives address APAs guidelines for undergraduate education. And then there is a rigorous review process where other experts in the field provide feedback and often suggest additional learning objectives

John: Was there much of a challenge getting agreement on learning objectives. I know in many disciplines, there’s a lot of discussion of what should be in the core material in the discipline? Has psychology been able to overcome that a bit by agreeing on these standards?

Alan: Well, I think it is a challenge in every case, because everybody has a sense of what they think should be taught and what is essential for students to learn. Like I said, I think it’s hashing out. It’s sometimes a little bit of trial and error, where we identify a learning objective, and then start writing about it. And then at some point, we may decide that’s not the right objective to have. But I think in the end, the subject matter experts and the content managers do a really good job of coming to some consensus on what we want to teach.

John: Was the content in the courses, the actual readings and other materials, developed by the people who created the package?

Alan: Absolutely. We are a digital first product, with original content developed by experts in various sub disciplines in psychology. And this really gives us a big advantage I think over print textbooks. So starting from scratch, in an entirely digital environment, we can develop a seamless presentation, where we start with learning objectives, we present content, we design activities and interactive exercises, and then have assessments. And we can be sure that all those things are aligned with learning objectives. And we don’t have to try and retrofit learning objectives into existing content. And then just hope that assessments that we have address the new learning objectives. So I think being able to work in a digital environment allows us to have this really seamless presentation where we start with learning objectives and we go all the way through assessment, and all those things match.

Rebecca: You’ve hinted a little bit about maintaining the content, given that it’s in a digital platform. But can you talk a little bit more about how you maintain that or what that review processes is or how that gets updated on a regular basis?

Alan: Well, that’s a good question. One issue for us is that we’re still so new, that we haven’t had semesters and semesters of feedback from students that we can use to make revisions. But what is really exciting is that whenever we decide that we want to make a change, we can do it in real time. Think about a print textbook, whenever they decide that they want something to be different, they might wait three to five years for the opportunity to make those changes. With us, since it’s digital first, we decide something needs to be changed or updated or revised and we can make those changes in a really timely fashion. And I think it really helps to keep our content current.

John: And if you find that students are not achieving the learning objectives, you have the ability, at least, to make some changes, either redefining the objectives if perhaps the objectives didn’t work too well. Or if the learning materials weren’t quite aligning with that. How often is that type of change done?

Alan: Well, so far, we’re still into the development process, that we’re making those changes just all the time. As the products get older and more mature, I think we’ll need to do that less. But again, we’ll still have the opportunity to do it as often as we’d like to.

Rebecca: John mentioned at the beginning of the episode about his interest in this project related to adaptive learning. Can you talk a little bit about the adaptive learning component of this project and the platform that you use.

Anna: The platform that we use is CogBooks, it is a platform, I think, is really unique compared to other textbooks, because it allows students to engage in their learning, and really direct their own learning pathway. And that’s based on their understanding of the content. So instead of going through a whole chapter, and then maybe quizzing yourself or testing yourself, after each content page, they get some formative questions and they get to make their own assessment. They get to say, “Okay, did I understand that? Was I not too sure? How did I do on those questions?” And then they can choose whether they want to get that support material or not, which we think of as like, “Would you follow up and ask your instructor a question.” That’s how we think of this support material. So if a student is very confident with the material, they’re essentially going to get a different path than a student who might need help in a couple of different areas or a student who might need help in a lot of areas. There’s just more examples or more practice. So it can look a little bit different for each student, which is pretty cool.

John: Linda mentioned earlier that psychologists have done a lot of research and cognitive scientists have done a lot of research on how people learn, how have some of those principles of how we learn been built into this platform?

Alan: Well, Linda is certainly right. There is a lot of research on learning science. And because we are psychologists, we’re familiar with the research, and use that knowledge to help students learn throughout the product. I think the centerpiece of our efforts to use learning science is the inclusion of metacognition, that is helping students evaluate their own learning, helping them identify what they know and what they don’t know. And when they identify what they don’t know, then they could spend a little more time learning that particular topic. Often students get to the exam, and they haven’t really done a good job of testing themselves, identifying what they know, and what they don’t know. And the first time they realize they don’t know something is when they get to the exam. And that’s obviously not the right time to get there. And so we’re able to identify ways that we can help them evaluate their own learning. We help them in this process by incorporating those learning strategies that we know are supported by research. We present concrete examples throughout. We have frequent activities and assessments. Students get practice retrieving content from memory. And these opportunities are spaced just as the research suggests they should be. They have interactive exercises. They get practice in the elaboration, or explaining content to themselves in their own words. And really the process of taking material and from the jargon in which it’s presented to explaining things in their own words, is a really useful process for students to engage in. And we could have a whole podcast just on principles of learning science that PsycLearn incorporates.

Rebecca: Can you talk a little bit about what the student experience is like for students that are using the platform? So Anna, you hinted a little bit at this with some of the adaptive learning components. But what does it look like from: I entered this space from the beginning, and now I’m interacting with this over time.

Anna: So students really, I think, approach PsycLearn differently than a printed textbook, again, not only just the fact that they would go through a printed textbook, and then maybe ask themselves some questions, probably skip over a lot of reading, because maybe it’s not very engaging or easy to get distracted with print textbooks as well. But one of the really interesting things is that they’re engaged with the content pages within them. So they’re not just engaged outside of content, like “Oh, refer to this video,” or “you can reference this activity” or “try some completely different platform to help with your reading….” but within the content pages, they get activities in which they’re asked to reflect on the course material. How does this connect to things you already know? So they get to respond to those things within the content pages. They get practice formative questions, formative meaning that it’s not just did I get this right? Yes or no. But they get feedback. So they would find out what a suggested answer was to a short answer question. They can compare that to their own. With a multiple choice question, they get follow up on why that is the correct answer, not just “Well, if I’ve missed this a couple of times, and they get multiple chances to respond, but they get some explanation about why that was the correct answer.” And so we have video questions, we have matching exercises, we have short answer, we have multiple choice questions, different drag and drop. So we’ve tried to just vary the kinds of interactions that students would have, the ways that they can practice answering questions. And this variety really helps maintain interest throughout. So they get to engage it in lots of different ways. It’s not just a rote memorization, but they’re actually applying the material as they go through each module.

John: So you mentioned formative questions. Are there are some assessment tools built in to evaluate students’ learning for grading purposes, for example?

Anna: There are. Yeah, so there are summative questions. And we try to have them associated with those learning objectives that Alan had talked about earlier. And so after maybe one, two, or a small group of those learning objectives are covered, there is a short sample, maybe 5…10 multiple choice questions where students have a few attempts, and we want them to get to about an 80% accuracy. And one of the nice things is if they don’t on their first attempt, they can actually go access those support pages then. So maybe they skipped over them earlier. And they thought, “Okay, I got this, it’s fine. I understood this page.” And then they get to these questions later on, these mastering the content and they’re like, “Well, wait a minute, I was missing some of this.” So they get another chance to go back to those support questions, and try it again. So then hopefully, they can get those right after three attempts. We don’t want them to be punitive. We want them to get engaged with the material and get to that mastery.

Rebecca: So now that you’ve talked a little bit about the student experience, can you talk a little bit about how this supports instructors? What kind of feedback is provided to instructors and maybe how instructors are using this in their teaching.

Anna: One of the things I really like, we all know, anytime that we can even maybe think back to our own experiences in classrooms, it’s a struggle getting students to read. But a lot of these classes, I mean, really all of our classes, if we’re assigning some sort of book, some reading material, we want them to read it. So how do we get them to engage? Well, that’s where the accountability comes in. So we’re giving them this tool, and then their instructor can see, “Okay, they’ve made it through this material, they engage with this.” So the instructor could choose to then reiterate that material in class, or they could choose to maybe take an active learning approach in class. And okay, now that you’ve had this background, everybody made it through, we’re going to apply it. And maybe not everybody got to 100% on every question, but maybe we got to about over 90% completion, which is what I would typically see in my classes. So you would have students who, even after a few attempts, would sometimes miss some of those mastering questions, but generally, it was pretty high. And so I could see that before I went into class. Okay, so students, as a whole, on average have got this. So I can take it a step further, where I can go back to those questions that were really challenging. So you get to track progress through each module, you can also dive a little deeper with individual students. So a student comes to you and says, “I’m really struggling,” the instructor can go in there, they can look at their pathway. Were they accessing the support material? Did they take those quizzes a few times? Where were they spending their time? Or if the instructor wanted them to cover it on let’s say, Tuesday night, were they doing it all Tuesday at 9pm? Right? Like you could actually see all of that. So you can kind of work with them like, “Well, I see you started this that night, let’s think about how we could maybe back that up. Like maybe we should pace this out at least a few days ahead of time, which is a really nice way to work with students.” And it also syncs with the LMS, the learning management system, so it really takes that burden off of instructors as well. So it’s talking to the platform that we get there. So we get this easy back and forth, which is really nice. So we can nudge students along with that, I would often encourage students to visit those support pages if they were struggling, like, “Look, I know this is really challenging,” or, “Hey, there’s more practice, if you just want a little bit more practice, you can go to some of those support pages to get that.” And the other nice thing is students can actually email right from there, which I think is pretty unique. So if a student emails me, I could see okay, they were having problems in statistics with variability and standard deviation. And it will take me as the instructor right to that page. And so then I can see that and I can say, “Oh, well, why don’t you try these examples here,” which was really nice. Instead of just a student saying, “I don’t know, I don’t get this,” they can let you know exactly where they’re struggling. So it provides a lot of options for instructors.

John: And do instructors get aggregate statistics on what areas students are struggling in addition to data on individual students?

Anna: Absolutely. So you can see that on the dashboard. You can see if there were chunks of mastering questions, for example, or a specific question that students struggled with. So you could choose to review that and maybe even go over some of those in class or go over specific topics in more detail.

John: How are instructors combining this with other face-to-face learning activities, for example, in face-to-face classes. As I understand it, most people use this as a textbook replacement, except a much more powerful replacement to a textbook. What other things will people often do to provide more of a sense of community or a social component to the learning?

Anna: Yeah, so, I think a lot of instructors are using it, just as you said, as a textbook replacement. And that’s how I have used it for statistics in particular. So it was taking the place of my other textbook, which actually was a print textbook that was online. And so I don’t feel like there were any losses. But we were gaining the ability to say, “Okay, we’re all on the same page. I know that everybody’s at least had some exposure to this. So now let’s talk about it. I can see where you were struggling. So I know we’re going to need more examples here. I know we’re going to need more follow up here.” Or “why was this question challenging? Why was this hard?” So I think drawing on those parts are how we can maintain those social ties and the social interactions we want whether it’s online, whether it’s face to face, or in any capacity.

Linda: I’ll just go ahead and point out as well that part of the content package that we’re making available to instructors who adopt, PsycLearn course material is a set of student activities that sort of exist outside of the platform itself and are available for instructors to assign or to bring to the class. They’re typically designed again to align with specific learning objectives and to give the instructor options for an in- class activity or an additional assignment activity. And that’s accessible to the instructor when they’re using the dashboard.

Rebecca: Sounds like some really great support materials. Can you talk a little bit about how this platform has affected equity gaps that we might be seeing?

Alan: Sure, I think there are two ways that PsycLearn is addressing issues of equity. First, we are designing PsycLearn to be culturally inclusive, that is we work to make sure that through examples, images, photos, figures, every student can see themselves in the product. Plus, our goal is to include the research of diverse psychologists and psychologists of color. And then I think the second way that we strive to be equitable is to design for universal access. We want the product to be accessible to all students, and so we make great efforts to do so. We use typography that’s particularly accessible. We incorporate transcripts and closed captioning, we have text-based alternatives for non text elements. And our efforts are well beyond what the minimum guidelines would be, well beyond what typical expectations would be, and I want to say that this is because we value this process so highly. We are keenly interested in making an accessible, inclusive product, and I think we’re really working hard to do that.

John: Have there been any studies of the overall efficacy of this program compared to alternatives?

Linda: Well, we have conducted some impact surveys, surveys of student users to gauge their impressions of the impact of PsycLearn on certain important aspects of their experience. Through surveying students, we looked at the impact on their knowledge gains, on their confidence in their ability to use that knowledge, and on motivation to complete assigned activities. And on the questions that we asked, to sort of tease out each of these thematic areas, we did receive very positive response. Now, these are, albeit, impressions of students. That’s sort of the first line of study available to us. And we feel pretty motivated ourselves by what we’re learning, by what we’re hearing from students. These surveys did also include some open text entry questions. And very interestingly, some of the criticisms that we heard were often this is a lot of time, this is a lot of work. But by the same token, we’ll hear in the very same response, but it’s very worth my time, and it really helps. And I feel confident when I go into the class, and I already know what my instructor is going to talk about, I feel more relaxed, I feel more able to pay attention. That kind of response has been very rewarding to receive from these student surveys.

John: With some other platforms, there have been studies that have looked at the impact on equity gaps, and they relate very much to what you just said, that basically, students come into our classes with very different backgrounds. Some come in with a very rich and strong background from prior courses, others come in with a somewhat weaker background. But to make it through an adaptive learning platform, the students who come in with less background can acquire mastery, but it takes a bit more time and effort. And the students who come in with a stronger background can race through a little bit more quickly. But much of the research does suggest that those equity gaps tend to be reduced. But it is at the cost of additional effort by those students who come in with a little bit less prior knowledge or prior experience with some of the material. And so I think that’s one of those real strengths of adaptive learning platforms compared to other formats like textbooks and lectures and so forth, where everyone is expected to move at the same pace. But as Chuck Dziuban, on a previous podcast has said, in a traditional course, everyone spends the same amount of time with the material, but they learn different amounts, when they’re working with adaptive learning platforms, they have to spend different amounts of time, but they all can achieve the same learning outcomes, they can all reach the same outcomes. But it may take more time to do so.

Linda: That’s certainly our hope, and some of what motivates the work that we’re doing. I feel like having respect for the time that our students need to spend is top of mind. We need to make sure that what we’re asking them to do is a good use of their time. And therefore this mode of delivery and the kind of content that we make available through the multiple activities and so on that Anna was describing. We’re giving our students opportunities to do that self testing that reiterative process and not simply trying to digest a narrative.

Anna: I always think back to a student I actually had in class a number of years ago, and this student went through the same class multiple times. And it was a class that was needed to get to other upper-level classes and was very frustrated, of course. And then we came to find out that this student in the multiple times of taking the class, the book was listed as required, but with either a digital or print textbook, you really don’t have any way to monitor that. You don’t know if the student’s using the book even if they have it. But this student had never got the book, just never occurred to them that, okay, this could be a really valuable tool. And so that’s really a benefit. I see. Like you were talking about equity, let’s give them this tool. But let’s make it something that’s valuable, that can really help them and if they need that added support, they’ll get it. But kind of level the playing field a little bit.

Rebecca: How do you see this platform evolving over time.

Linda: So we are able to work fairly closely with our platform partner, CogBooks. We’re able to contribute to their roadmap planning to some degree. We’ve collected feedback through a variety of channels, also, not only the student surveys that I’ve mentioned, but we also, in the past year, conducted a qualitative study that included student interviews and observations of students who were assigned to use PsycLearn. And this exercise gave us really valuable insight into how and where students focus their time and attention. And so for instance, through that process, we were able to get some real good insight into how they approach the summary of each module. And as a result, we’ll be able to give some greater attention, we’ve got some plans to enrich and make more accessible, the content that we placed there, because we were able to learn that they highly value that for later test preparation. So that’s just an example. We’re hoping to be able to continue to do those observational and interview studies.

Rebecca: So I’m really curious for each of you to answer this question. But what makes you really excited about PsycLearn?

Anna: I’m happy to go first, because I am a very energetic instructor. I will be at the front of the room, and I bring a lot of energy. And what makes me excited about PsycLearn is just the ability of students to be active consumers of their learning. I would love for them to take away something that is memorable, that feels unique. And so PsycLearn frees me up as an instructor, that I don’t have to go through every bit of background information in the way I use it in class, I can assume they’ve had things. Let’s spend time on things that are interesting. Let’s dive deeper into things. Let’s do more exercises for things that are challenging. And so the students get to think about and apply new ways of learning and engaging the material from the very first moment that they encounter the content.

Alan: Yeah, for me, it’s really about the inclusion of metacognition, I think it’s really helpful to have students be able to evaluate their own learning. I think this makes them do better on exams, and I think it makes them more successful in their entire college career. It’s also an important part of my own teaching. I have a lot of activities in class where students can answer and identify how well they know what they’re supposed to know. And what I often find is that students study, but they don’t know how much to study, they don’t know: “Do I know it as well as I need to know it.” And this product gives them a lot of feedback, really helps them understand if they understand the material as well as they need to. And PsycLearn really can do that in ways that regular textbooks cannot.

Linda: I would say that my excitement really comes from the big picture, the opportunity for us to deliver on these experiences that Anna and Alan are describing. But we’re going to be able to do that across the foundational courses in the curriculum. And so that larger picture future horizon is pretty exciting.

John: Well, you’ve already addressed some of this, at least in terms of PsycLearn, but we always end with the question: “What’s next?”

Linda: Well, we have sort of talked a little bit about future and what’s next and our opportunity to be iterative. The digital platform allows us to be continually rethinking what is the best activity to offer to reinforce or bolster certain concepts. And so that’s certainly something that is ongoing. We can take feedback from students on how well they feel various content is supporting them. We’ll be looking at what we can do to encourage a greater level and engagement with that support material, so that students aren’t feeling like, “Oh, that’s extra, I don’t really need to go there.” Indeed, it’s extra. But it’s really vital, especially for students who might sort of marginally understand. And so the opportunity for us to deliver a reframe of a concept can make all the difference at that tipping point.

Anna: Yeah, and I think what Linda had said about using the support material is really important to us. So we’re really interested in applying these learning science principles that Alan had talked about to say, “Okay, what is the best way to get students to engage with this.” We want them to see that cool stuff we’re putting on those pages, if they feel like they could benefit of it. So we want to reach the students that need that support. So I think just diving in a little deeper and figuring out the best ways to reach them, to get them to say, “Okay, yeah, I could use a little practice.” Or “I could use another example,” snd really assess that learning of their own. It’s a challenge. It’s a challenge we have constantly, especially when students are new at something, they tend to think, “Oh, yeah, I got this. I know it all.” And so getting better at that metacognition, I think, is really exciting to us.

Alan: Yeah, and I think for us, it has a lot to do with making a better and better product. I think about what it was like the first time I taught a course. And I equate that to the first iteration of a PsycLearn product. And then I think about what was it like the fifth time I taught that course or the 10th time I taught it, and how richer and deeper it was. And I think the same thing will happen with PsycLearn. We’re going to keep on creating a product that gets richer, more inclusive, and more likely to help students succeed.

John: I was really excited to hear about PsycLearn, and everything you’ve talked about makes me even more excited about the opportunity for this and I hope that other disciplines will start working on similar materials for their basic courses, because the benefits from this have been well established in terms of improving student learning and reducing some of the equity gaps. Thank you.

Linda: Thank you.

Alan: Thank you.

Anna: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

250. Hacking Assessment

Traditional grading systems often encourage students to focus on achieving higher grades rather than on their learning. In this episode, Starr Sackstein joins us to discuss how classes can be redesigned to improve student engagement and learning. Starr has been an educator for 20 years and is currently the COO of Mastery Portfolio, an educational consultant, and instructional coach and speaker. She is the author of more than 10 books on education, including the best-selling Hacking Assessment: 10 ways to go gradeless in a traditional grades school, which has just been released in a new edition.

Show Notes

Transcript

John: Traditional grading systems often encourage students to focus on achieving higher grades rather than on their learning. In this episode, we discuss how classes can be redesigned to improve student engagement and learning.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Starr Sackstein. Starr has been an educator for 20 years and is currently the COO of Mastery Portfolio, an educational consultant, and instructional coach and speaker. She is the author of more than 10 books on education, including the best-selling Hacking Assessment: 10 ways to go gradeless in a traditional grades school, which has been released in a new edition. Welcome, Starr.

Starr: Thanks so much. I’m excited to be here.

John: Today’s teas are… Starr, are you drinking tea?

Starr: I am drinking water. No tea unfortunately, not yet.

Rebecca: Not yet. Okay. See, there we go, there’s promise there. I have Scottish Breakfast tea today.

John: And I have spring cherry green tea.

Rebecca: Well, that’s good.

Starr: Those both sound delicious, really.

Rebecca: So, you haven’t had that one in a while, John.

John: I haven’t had any in a while…

Rebecca: true that…

John: …we took a pause in recording for about a month. [LAUGHTER]

Rebecca: Fair. But even prior to that it had been a while I think.

John: I think so too.

Rebecca: We invited you here today to discuss Hacking Assessment. The first edition of your book seven years ago helped to launch the ungrading movement. Could you give us some background on what prompted you to move away from traditional grading systems?

Starr: Absolutely. In years one to five when I was in the classroom, I would say that I pretty much did grading and assessment the way it was done to me. And the one major significant thing that changed during that time was I had a child. And in his elementary school, they actually use standards-based grading. And when I got his first report card and saw just how much information I got from his teachers, and how the behaviors were separate from the actual learning and the narratives were really aligned with where he needed support and what was going on. I was like, Mmmh…for someone teaching AP English, only having the opportunity to give one grade, with pre slugged sort of comments that I was allowed to bubble into my… back then we were still using Scantrons for entering grades. I’m definitely dating myself by saying that, but it’s the truth. And I started getting really frustrated with that. And from there, I started doing a lot of reading. Alfie Kohn has really played with a lot of these ideas for a long time now. And then folks like Ken O’Connor, who had the book 15 Fixes for Broken Grades, his first edition, I think it’s been republished twice already, in the time since I’ve read from there. I read his book, and I was like, “Oh, my God, I am doing all of this wrong.” There are so many things on this list that I do, and I never thought about it that way, and it’s just not how I want to keep doing things. And I think there’s a synergy with when you decide to read a book, whether or not it resonates with you and whether or not you’re ready to start implementing the things that you learn. And I think I was very ready to first acknowledge that the practice I was doing wasn’t serving my students as well as I could. And I was looking for alternatives. So having those jumping off points, having read a bunch of different things, and then meeting Mark Barnes along the way as well, and experimenting with alright, well, these are suggestions for this kind of space. What does this look like in New York City public schools, as an 11th and 12th grade English teacher and also as a journalism teacher? How do I start making this work? And that’s sort of how it all happened and then it took years to figure out how do I make this work well, because I did it for a while before it worked well. [LAUGHTER] There were a lot of mistakes, unfortunately.

John: We’ve been dealing with a number of people starting to experiment with ungrading in college, but it’s a little bit easier in a college environment, I think, to make these changes, because there’s a little bit less structure imposed on teachers. How were you able to implement this in a K through 12 system?

Starr: So I think I was very fortunate to be in a very small community when I started doing this. We were six to 12. I was already a very established teacher in that community. I had a track record of getting students prepared for college. And most of the families when I made choices, always kind of knew that they were intentional, and there were reasons. And in my AP classes, that was probably the most struggle, because parents get nervous when they have 12th graders, what is this gonna look like on the transcript? How is this going to impact my students moving forward from school? And I just really tried to set up systems and to be super transparent about everything that we were doing so that first of all, I live streamed my class a lot, for better or for worse. And I say that because not every class was a winner. So if you were watching when it wasn’t a winner, like, well, this is reality, it wasn’t a good day. But I think they were able to see the rigor of what was going on in the space and despite the fact that it didn’t look like what normal AP classes looked like, they could appreciate my wanting to be flexible to the individual learners in my classroom… that even the creative projects I was asking them to do was often a lot more intensive than just doing a test or just writing a paper and gave that level of inquiry into that process as well so that students could be really excited about the learning they were doing. And the more comfortable I got with different technologies… I experimented with blogging to increase reading. That’s one of the biggest problems in English classes. I think most kids don’t read the books for a lot of different reasons. So how do you get them to read when you’re teaching a literature class, beyond just the five or seven or 10 books you’re reading as a whole class. So they started blogging, and we started using the blogging communities for recommendations on different books they were enjoying on their own and why they enjoyed it. And I really encouraged them to use that space too as a way to develop their writing voice. So it wasn’t like analytical writing all the time, it was more conversational… reaction sort of stuff to what they were reading and focused instead of like overviews of everything that they read… an analysis paper, which isn’t always fun for every single kid. I started tweaking that and I think parents appreciated my transparency. I did screencasts of our dashboard, because I had changed the way I was using the tool that my whole school was using. So like, if you have any questions, this is what it looks like, this is what you’re seeing. And if they emailed me, I just really tried to get back to them immediately, so that I could really put their concerns to rest before they started doing the thing that parents do, where they start making it a lot worse than it actually is. So I tried to catch that right away. To be honest, though, my colleagues were the ones with the greater pushback than parents and students… a couple of students, but just shifting the conversation away from grades, instead of what did I get? What did you learn? How can we track that progress over time? How do you know you learned it? Where do you see that evidence in your own learning. And I think very soon after getting in the routine of this is how we do things now they got it and saw that the level of metacognition as well as the rigor in the actual tasks were much greater than what they would have been seeing in a regular class anyway. So sometimes I got the: “this writing reflection is like a whole other paper that you’re asking us to do.” And I’m like, “Yeah, it is. But it also helps me give you better feedback, and it also helps me know where I need to adjust my instruction. So there’s a reason and it’s worthwhile, and it’s gonna help you, when you’re not just in school. This is a practice that you’ll probably carry with you.”

Rebecca: One of the things that you just brought up, Starr, is something that I definitely want to follow up on, which is getting our colleagues to also buy into this, and administrators. We exist within systems that require grade inputs. Grades are transcripted. So how do we get the people around us who support us professionally, to get on board? And what does that actually look like, functionally, when we’re generating grades when we’re saying we’re kind of ungrading all semester.

Starr: So those are really good questions. And in the second edition, I actually have built in leadership tips to support leaders who are unfamiliar with this kind of assessment practice and how they can support teachers who want to do it, if they’re not doing it wholesale as a school. I advocate for systemic use of this practice, because if we catch kids much younger, by the time they get to high school, their language and fluency in discussing their own learning is a lot greater. I was a 12th grade teacher, my kids had come through an entire system where this was not how it was done. So it was like, literally at the last minute, I’m like, “Yeah, I know, that’s the way you’ve been doing it this whole time. But we’re gonna do it a little differently and I promise you’ll still get into college.” It’s a different vibe. And my colleagues, I think, knew my students appreciated it, because they would start hearing from my students: “How come you don’t do this?” …which is also like a little bit of a target was put on my back, because if a school or a district is going to make the shift, it requires a lot of professional learning. And if you aren’t the kind of teacher who makes the time to do learning on your own, then there really does need to be supports put in place prior to it’s happening. And I’m a super reflective teacher, I did National Board Certification, I will go out of my way to get myself to a conference even if my school wasn’t paying for it. Because as an educator, I felt it was an essential part of my job to continually grow and model that for my students. But not every teacher is like that. And I’m not suggesting that everyone has to be or whatever their process is, but I do think it’s important to invite colleagues into your space, give them that “what you could do tomorrow kind of tips” like what are the first few steps you could take to try this out before you commit to it wholesale. And in terms of the grading aspect, the way that I got around the traditional grading was assessment conferences with my students. So really building in a vibrant and robust portfolio system where students were collecting their learning over a larger period of time, giving them the vocabulary to talk about their growth as they looked at those things, and then a conversation just like this. So based on the standards we worked on this marking period, where do you find yourself in terms of mastery? And what does that translate to for a report card grade, because I had to put a grade on the report card as well. So it was really just making them acutely aware of what exemplary work looks like, how they were meeting benchmarks to get there over time, and then also switch that transactional sort of relationship around getting grades to a more progress minded model, where they understand learning doesn’t happen in one sitting. And even though you may have successfully completed one assignment, that doesn’t mean you’ve mastered a particular skill, it’s just your first go at it. In order to get to that mastery level, you have to do it over time with less and less support, and kind of do it on your own.

John: What sort of buy in did you get from other teachers that you were working with?

Starr: It was secret at first. There were like people just dropping by out of curiosity to see what was going on in my classroom. Then a couple of other people just asking, “what would this look like in my gradebook?” I was very lucky in the one sense that our whole school was a portfolio school. So that part of it was already there. And then I also did some PD with my colleagues around reflection practices. We tried to really create something that was consistent, and also the same. So like I had created a process for doing reflection, which is that five steps sort of: first, you have to reexamine what was it I was asked to do? What were my steps for completing the assignment? Where do I think I’m meeting the goals that I set for myself? How am I doing that? What level am I doing that at? And what would I do differently in the future? And then we kind of scaffold that down to sixth grade up to 12th grade. So what is that kind of reflection look like in a sixth grade classroom, a seventh grade classroom, all the way up to 12. So that there are realistic expectations in that space around those things. And my classroom was always open. And I resented the fact that when my principal decided that she wanted us to go to a standards-based model, I implored her to not do it the way she did. I think we should have a pilot team, we should have a committee that does this, we should test it out first, try to get either a grade level team or a content area vertically to commit to doing this and then have input from more people. And then we need to train folks in the areas they aren’t already familiar with, starting with unpacking standards and getting them comfortable with that kind of language and what our expectations are. But that’s not what happened. It was like an email that went out. We’re going to do this this year. And it was a disaster. And I got attached to the disaster as a direct correlation to how all that happened. And unfortunately, you get one good shot to make a significant assessment or grading shift in a decade, because unless your folks are leaving quickly, no one forgets. So really setting up systems in the future, if folks who are listening want to do this on a bigger scale, set yourself up for a three- to five-year implementation plan, start small and grow it organically and provide tons of support along the way so everybody feels confident and not just your teachers, your community also. What does this look like for your parents? What are they going to be receiving that’s different? And just make sure that you have answers to commonly asked questions on the front end, so that when new stuff starts coming in, you’re ready to triage that, you’re not just answering the standard questions over and over and over again.

John: You mentioned in your first edition of the book that one of the motivations for this was to get students to focus on their learning rather than on grades. How successful was this? Did this work for most students?

Starr: For most, yes. And believe it or not, the ones that don’t traditionally do school well, who don’t play the game, it worked best for them. And as three educators sitting on this podcast right now, I think we can all agree that sometimes our brightest students are not the ones who do the best. The ones who do the best are the ones who are most committed to getting high grades and kind of checking the boxes and doing everything that they have to be compliant for in order to get that score. So when we shifted the focus away from that and started looking at skill acquisition and content deepening, and really getting them to be able to advocate for their own needs in that specific area, I think that it wasn’t just about them completing the tasks I asked them to do, but it required them to engage with me in a dialogue in the kinds of tasks they wanted to be doing, the way they wanted to be doing it. And it required my flexibility with taking that input and actually putting it into action. So I think that once they saw that I was listening to their feedback actively and using it right away to shift the way class looked, they understood that I wasn’t just saying, “I’m asking you to do this,” it was a real partnership, where if this is going to be successful, and you want your voice to be heard, you need to contribute or else you can’t complain when you don’t like what ended up happening, because I really did try to say “yes,” just about to everything, if they could articulate how their decisions and their choices aligned with what the objectives were, then I was totally hands off in their process to sort of help them be successful in the big picture. And it also really decreased the amount of folks who didn’t participate in the group work or didn’t participate in the learning. So when people say my students don’t finish work, or they don’t submit things, to me, that’s a red flag that either something else is going on that you need to get to the bottom of, or the kind of learning you’re asking them to do isn’t resonating. And rather than just pulling out the binder from what you’ve done for the last 20 years, you really do have to make a concerted effort to make changes so that it meets the needs of the kiddos that are sitting in front of you right now.

Rebecca: So you’ve talked a lot about reflection, and the role reflection is playing. Can you talk a bit about how you were able to get students up to the level of reflection that is really meaningful and gets to this metacognitive skill, building

Starr: Feedback, feedback, feedback. We give a lot of feedback to everything that kids do in the classroom. But the first few times we ask them to reflect, it’s so important that we’re also giving them feedback on their reflections, providing exemplars for them, really creating success criteria too, like that co-construction, like if I’m telling you, these are three examples that are wildly different, but all successful, what do you notice about all three of them? What are the things that need to be a part of every single reflection that we do. And then as they do them, rather than have them revise every single one that they do, since they’re doing them with every major assignment, it’s like, “alright, well, now take the feedback you got from the last one, apply it to this one and let’s see if we can’t grow you.” And usually by, I would say November, they’re already writing fairly good reflections and their ability to have conversation about their level of learning already starts to increase, because by November, you’ve already had a progress report conversation, you’ve already had a quarter one report card conversation. And I was doing a lot of modeling myself, like I would reflect openly on how successful projects went, in my estimation, and be really, really tied to the outcomes. And not just what I think or what I feel, but what I noticed, and how I would do it differently if we had the opportunity to do something similar again. And I think, again, that level of transparency and my comfort with saying to them, I don’t know how to make this better. What do you think? What made this experience challenging? Were my directions not as clear as they could have been? What do I need to learn from this experience? So it was very much a two-way street, which took time. And I do want to say that too. Like, I think I was seven or eight years into the classroom before I was comfortable enough to say “I didn’t know something.” That takes confidence in a way that you don’t really think. In the beginning of my career, I felt like I needed to be the expert over all of the students in my room, and I had to have an answer for everything. And I said a lot of wrong things because I was trying so hard to look like an authority. And I think the older I get, the more I work with educators, the more I realize that I’m a learner, I don’t know everything, even the stuff I’ve spent a lot of time teaching I don’t know everything about and new perspectives are incredibly useful in how I approach something because it’s the first time this group of kids is seeing something I might have tried before. Their input is extraordinarily useful for me to make changes moving forward.

John: It’s also a great way of nurturing a growth mindset in students by reminding them that we’re all part of this learning experience together. And that no matter how much experience you have, there’s always more you can learn. And so I think that’s a really great process. And it’s something that I think it generally takes a while for most people to get to.

Starr: Yeah.

John: So you mentioned having conferences with students, how often do you conference with students?

Starr: So, there’s lots of different levels of conferencing. So you have your in-class formative conversation where they’re asking questions and you’re taking the pulse of whether or not you’re going too fast or if you need to stop the class and do a mini lesson on something you notice everyone’s struggling with. Or if you pull a small group because only a small group of kids are really having an issue. So there’s that kind of on-the-fly conferencing where you’re walking around with a clipboard or an iPad and you’re taking notes on what you see. And then listening to the questions kids are asking and making a determination as to whether or not this is a small or bigger issue that needs to be addressed. And then there are formal conferences where kids are coming prepared to have that conversation where you’re giving them time in class. So part of my structuring… because remember, I said it took me a long time to find a system that worked that ended up in Hacking Assessment… so I started creating Google Forms, where there were very targeted questions that also aligned with the assessments that we did, and the different pieces of learning and the standards that we were addressing at that time. And before they could set up a conference, they needed to fill in that whole Google form, then I had all that informatio, so I could really target clarifying questions or gaps that we could spend our five minutes talking about. If they had done all the work to do certain things, they don’t have to rehash what I could read. And if I had 34 students in most of my classes, so there’s a lot of kids, there’s a little time, you really have to make that three to five minutes count, and give every student the opportunity to give you the most information that you could have to be able to determine what was going to go on the report card. So those conversations certainly got a lot better over time as well. The first one, there was a lot of prompting from me, a lot of questions to get them ready by conference number 2, 3, 4, and certainly by the end of the year, if you watch on my YouTube channel, I have examples of what those look like. By the end of the year, the student is doing 98% of the talking. And I’m just redirecting if they kind of get off a little bit, or if they miss a spot versus at the beginning, it’s more of like a 40-60 where I am interjecting and kind of bolstering confidence, helping them set goals and stuff. So there’s more of a give and take at the beginning of the year.

John: You mentioned giving students some choice in terms of the assignments and so forth. What are some of the more interesting assignments or learning activities that your students have come up with?

Starr: The one that always comes to mind was, towards the end of my time in the classroom, before I became an instructional coach, I literally gave my students my entire unit plan for Hamlet. And I said, “Alright, this is the way I always teach it. But I want to do it differently this year. So I want you to look at the overall objectives. And as a group, I want you to come up with something different, then we’re going to vote as a class, which group suggestion we want to go with, and whichever group is chosen, you’ll come meet with me at lunch, we’ll design an assignment together and work through the success criteria and benchmarks for doing it successfully.” And if I tell you some of the things these kids came up with, I would have never come up with in a million years. And what we landed on was these psychological profiles of the characters of Hamlet, where they had to first use the text, to use Shakespeare’s language, to diagnose them with some kind of psychological issue. For example, Gertrude would be a narcissist. And then they do research on the actual issue, so there’s a research component as well. And then they had to come up with a treatment plan for the character and create a movie that demonstrated the growth from whatever the treatment plan was. And what it really did was have this really in-depth character analysis of each character from Hamlet, regardless of which character you did, you were set on a course. And then we also created this Google form, so that when we had screenings of the movies at the end, students were actively taking notes about what they learned about the characters and giving feedback at the same time to the creators of those movies about what they learned and what they were still curious about. And it was really phenomenal, honestly. I think that I wish I would have started doing stuff like that sooner. Other examples would have been students creating movies in Minecraft, like for our satire movies, that’s usually so like, just technology, but I was very uncomfortable with, that they were able to use that. I was like, yeah, “If you could do it without my support, I could help you with content, but you’re on your own for the technology.”

Rebecca: So you’ve hinted at some of the changes in your second edition. Can you highlight some additional changes between the first and second edition?

Starr: Okay, so yes, there are a lot more resources. So over the last seven years, part of the reason I hadn’t made a second edition up till this point, was because I really wanted there to be a value added. I wanted there to be new voices I can highlight. I was really also looking for systems that started doing this work because I wanted there to be more case study material that kind of went in that it wasn’t just single teachers kind of playing with it, but actually systematizing it in ways that work for them. So there are brand new hacks and actions for every single chapter, all of them have read the first edition and implemented it in their own way. So what you’re getting is people’s take on how what they learned looks like. I really tried to implement K to higher ed. So Susan Blum did write a section as well on what it looks like in college for all of my reticent K-12 folks who were like, “This isn’t going to be viable in the future.” I had central office people write about stakeholder buy in and how they brought this into their space from a leader perspective, instead of just a classroom perspective. A lot of new tools that have been developed in the last seven years, lots of stuff about that, rubrics, progressions, not just in English, which was my background, obviously, really trying to span math, science, social studies, related arts. So there’s one with a music teacher writing about how they’ve done that in that area… elementary teachers. So there really are tons of resources with a lot of different fresh voices who are using this now, as well as a very intentional talk about equitable practices. I think a lot of this stuff is equitable, but I never thought of it in that lens until COVID. And then once COVID happened, really trying to talk about how these things address some of those gaps that need to be addressed, but weren’t explicitly tied to them in the past. So that’s really where the bulk of things have shifted. And then there’s an incredible appendix with lots and lots of examples of everything.

John: And your first edition was wonderful. It provides a lot of good resources. And in each section, it talks about how to deal with pushback, which is one of the things anyone introducing something new has to deal with. So I’m assuming that continues into the second edition.

Starr: Yep, sure does.

John: So your first edition was very successful, and has received a lot of traction at all levels of education, and helped spur the ungrading movement at the college level that we’ve been talking about a lot in the last couple of years with our guests, and with many of our colleagues. For those people who have read the first edition, what would be the benefits to them of picking up the second edition, and who should they share that with at their institutions?

Starr: So I’m really hopeful that this time, it’s not individual teachers picking the book up on their own, although I certainly advocate for that. I want to see teams use this as a PLN opportunity and explore the text in a way that makes sense to them. It is not narrative, necessarily. So each chapter is its own sort of entity. And so I would encourage folks to choose the chapter that they’re most ready for at this moment and pick it apart in a way that’s going to make most sense for their practice.

John: We always end with the question: “What’s next?”

Starr: Oh, I’m so glad you asked. So what’s next for me right now, we are doing a free book study with the new book when it launches August 2, and it’ll be on Amazon. And then also, once this one launches, and things are moving, I’m under contract with ASCD for my next book, which is specifically about portfolios and student-led conferences. So that is still something that’s a little thinner in Hacking Assessment, because I think that that really requires a little bit more depth than I could give it in that book in one chapter. So I am currently working on that and really trying to gather with some of the districts that I’m working with to build really great systems for building portfolios. What does that look like? And how do you parlay that piece into these student-led conferences so that you can have a robust system in your space?

John: That sounds like a great supplement. Well, thank you. It’s great talking to you. We’ve heard mention of your book from many of our past guests, and I’m glad I was finally able to get to read it. And I’m looking forward to the second edition, which should be arriving soon.

Starr: Awesome. Thank you so much.

Rebecca: Yeah, thank you so much. This is such great information and we’re looking forward to all your new work as well.

Starr: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

246. Embedding Career Competencies

Students generally enter college to advance their employment prospects. In this episode, Jessica Kruger joins us to discuss how explicitly embedding career competencies in the curriculum can engage and motivate students. Jessica is a clinical assistant professor in the Department of Community Health and Health Behavior and is the Director of Teaching Innovation and Excellence at the University of Buffalo.

Transcript

Rebecca: Students generally enter college to advance their employment prospects. In this episode, we explore how explicitly embedding career competencies in the curriculum can engage and motivate students.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist&hellip

John: &hellipand Rebecca Mushtare, a graphic designer&hellip

Rebecca: &hellipand features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Jessica Kruger. Jessica is a clinical assistant professor in the Department of Community Health and Health Behavior and is the Director of Teaching Innovation and Excellence at the University of Buffalo. Welcome back, Jessica.

Jessica: Happy to be back.

John: We just saw you at CIT.

Jessica: &hellipgreat conference.

John: It was nice seeing everyone back in person again. For me, and I think for Rebecca too, this was our first conference in person in at least a couple of years.

Rebecca: Yeah. Yes, it was. Refreshing.

John: Speaking of refreshing, our teas today are&hellip Jessica, are you drinking tea?

Jessica: Iced Tea.

Rebecca: &hellipthe best kind during the summer.

Jessica: Exactly.

Rebecca: I have some nice piping hot Ceylon tea again.

John: And I have spring cherry green tea.

Rebecca: Is that a new one for you, John? Or you just haven’t had it in a while?

John: I haven’t had it in a while.

Rebecca: That sounds good&hellip

John: It is very good.

Rebecca: &hellipif you like cherries, which I don’t. [LAUGHTER]

John: It’s spring cherry, it’s not just cherry, these are spring cherries.

Rebecca: &hellipthe best kind.

John: &hellipfrom the Republic of Tea&hellip actually from Harry and David. But it’s produced by the Republic of Tea for Harry and David.

Rebecca: So we’ve invited you here today to discuss your work in incorporating career readiness into the public health curriculum. Why should there be an increased focus on incorporating career readiness in our courses and degree programs?

Jessica: I like to tell people, it’s not just our job to teach students really cool things, it’s our job to help them get a job. And so ultimately, by incorporating career readiness skills, we’re equipping our students to go out into the world and get their first job. And really, in public health, there’s so much work to be done. And so if students aren’t ready to do that interview, or send out their resume, or even talk about their experiences that they’ve gained in the classroom, they’re not going to land that job.

John: Has it been hard to convince other faculty of the need to provide this career readiness for students?

Jessica: So myself and another faculty member have been incorporating the career competencies in our courses, we both teach a 200- and a 300- level public health course. And, in those, we are getting students anywhere from sophomore, juniors, or seniors. And I think it makes sense in our profession where many students go out and get jobs after their bachelor’s to have them start thinking about this early, even as early as their second year in our courses.

Rebecca: Can you talk a little bit about the kinds of careers that your students in your program pursue?

Jessica: Yes, so we have students working all over. And I’m really proud to say we have a student who’s actually working at a safe injection site in New York City, which is one of the first that has been opened. We have students working locally in health departments, working for hospital systems, and local nonprofits. And so our students are really going out there and doing what needs to be done, especially what we’ve seen over the past two years with the pandemic. We’d had so many students work in contract tracing, and also with local health departments, and it’s growing. Public health is becoming more recognized. And that’s why we need to continue to prepare our students for what’s next.

John: For those listeners who aren’t familiar with safe injection, could you talk about what that is?

Jessica: The first facility was opened in New York City. This allows for individuals who use injectable drugs to go into a place that is clean and monitored and inject safely. Someone is there to monitor them, provide them clean supplies, and even help them if there is an overdose. So this is a harm reduction technique that prevents deaths.

Rebecca: Can you talk about the ways that you have employed career readiness into your courses?

Jessica: We use the framework of the NACE Competencies and NACE stands for National Association of Colleges and Employers. And these are competencies that were set forth by actual companies, employers out there, saying what they actually want in a new graduate. So there are actually eight different areas, things from teamwork, leadership, ensuring that students are able to be critical thinkers, have career and self development, include equity and inclusion, and be proficient and technology. And so what I’ve done is incorporate first starting at the syllabus level, incorporating some verbiage saying: in this course, you’re going to learn career competencies, and we’re going to cover seven out of eight of these career competencies. Because I’m teaching a 200-level course, I’m not focusing on the skill of leadership, that’s a little bit higher level that I’m working at in this course. But in every assessment that I have in class, I have not only the objectives, why we’re doing it, but I include what competencies we’re working towards, and how they can talk about this in a job interview&hellip to a internship site. So it’s not just that you’re writing a paper, you’re working on those written communication skills, and you’re able to articulate that

John: How have students responded? I would imagine it would increase their motivation a bit when they see how directly applicable these skills will be for them.

Jessica: It all comes down to transparency, because the more transparent and applied students feel that their assessments are, the higher quality I tend to see their work. And so by telling them, you’re not just writing a paper for me to read, you’re writing a paper to practice this skill. And you’re also going to do a presentation, because in public health, you need to have excellent written and oral communication skills. And hey, you’re also going to make a poster on Canva, because you need to know how to use technology. And so by kind of stringing these competencies together, it allows students to see that what they’re doing is not just for a grade, but to help them build those skills.

Rebecca: Can you walk us through a specific example?

Jessica: Yeah, in this course, I have students do a variety of writing samples on different problems. So, in public health we’re very applied. In one of the papers, students are talking about how public health has been influenced by other areas&hellip philosophy, psychology&hellip and so they could just see it as a paper, a 1.5 page paper, or thinking about it from a career readiness standpoint, they’re learning how to write succinctly and to whatever audience&hellip so, in this case, a lay person learning about public health. And so in writing this, I include why they’re doing it. And when I’m giving the example of what I’m looking for in the assessment, I often have the students reflect on what they’re doing and why they’re doing it. I think it’s important for them to think about that, why am I writing this paper? Why are we doing this? &hellipto improve your skills, to work on that written communication. Another example that I think is probably maybe a little bit more applied here is students write a paper, they record a short pitch of this paper using Flipgrid. So they’re doing written communication, oral communication, and in that little flip grid that they’re doing, they have to dress professionally, practice that skill, and give feedback to peers. And so all of that together, which is seemingly not a too arduous of an assignment, is really hitting on a lot of those career skills.

John: Have other faculty in your department picked up on the use of expanded career readiness in their classes?

JESSICA :Yes, one other faculty member has joined me and currently we are studying the effects of this on our students. So what we have done is ask our students at the end of the semester to complete a short survey asking them about how they felt about including career readiness into their courses. And overwhelmingly, students are so appreciative that we’re thinking about this, preparing them for what’s next. And it also shows that we care, we’re not just there to be a sage on the stage. We’re there to help them get ready for that job, that grad school, or whatever else they choose.

Rebecca: Do you have other findings from your research so far? I know you’re early in this process.

Jessica: Another major takeaway from this research is students wish they had this much earlier. They wish that we started talking about it day one. And while we could have told them about the career design center on campus, they’re not forced to practice this in their courses. They’re not tying this to that NACE Competency framework. But really, when we think about it, career readiness is everyone’s job. In our University of Buffalo, we’re working to create a career ecosystem, meaning that faculty are really on the front lines of this. They’re the first people that students are going to go to for career advice. And not all faculty are equipped to have that conversation. But at least, at the very minimum, being able to direct them to the Career Design Center and other resources that are available through the Career Design Center, I think is key. One simple way that I’ve introduced students to the actual physical Career Design Center on campus is I have them go take a selfie. They have a great little selfie station, and that’s one of their first things that they do in my course. So I can start to put a face to a name. And they can also learn about why it’s important to have a professional selfie. So they can put on their digital profiles like LinkedIn.

Rebecca: I found in some of the classes that I teach, I’ve in the past done assignments where we do professional email communication related to the work that we’re doing, or reports that might be common to the discipline. And students have responded much more positively to those kinds of writing assignments, because they can see the practical application and can connect the very specific, like, I can see how I’m doing this skill, and it’s gonna result in me being able to do this other thing. We just don’t always articulate that when it maybe is a little more abstract, when it’s maybe a more traditional paper and how that might tie to the kinds of work that they might do as a professional.

Jessica: Most definitely. And I’ll say this really was not much effort, I did not change what I was doing in my courses. I really just added a sentence or two to each of my assessments, relating it back to the NACE Competencies, I also brought a little bit more awareness to what we were doing in class. When we would do group work, I’d say, “Oh, you know what? You’re working on teamwork. That’s great, because in public health, we work in teams, and this is also related to the NACE competencies.” So it doesn’t have to be like a capstone course where you’re deliberately working on resumes or other career competency. But I think weaving this in, adding to it, and really raising awareness about some of these skills can really help students go to that next level, or start thinking about what they need to do to build some more skills before they leave our universities.

John: You’ve already answered this partly by talking about teamwork and other skills that can be done in any class. But what about those instructors in a course on abstract algebra? How would you build in, say, career readiness into that, other than the types of things you’ve already mentioned?

Jessica: Well, I think teamwork and technology would be two very easy competencies to weave in to any sort of course. Also, if you’re doing advanced level math, you’re probably using critical thinking skills. And you’re probably talking about what your findings are. So you’re hitting communication, you may not have much on professionalism, or career and self-development, but still, you’re hitting five career competencies, and not having to add anything, but really just highlight what you’re already doing.

Rebecca: Were you surprised when you sat down and looked at your syllabus and your learning objectives and looked at the NACE competencies and put them side by side and how well they aligned?

Jessica: Yeah, and in fact, it’s interesting, because in public health, we have an accrediting body called CEPH. And our CEPH Competencies for preparing students at the baccalaureate level actually align perfectly with NACE. And so it made sense, once you sat it down to say, “Oh, I’m already doing a lot of this.” And yeah, it might take part of my class time the second day of class to bring in one of the career designers so they put a name to a face, but it’s not taking away any time from my instruction. And by adding the transparency in that connection. I think it’s created more of a caring environment for my students. They know I care about what happens to them after this course. And more students have come to me than ever, asking about what’s next, whether that’s grad school, or how to apply for that first job. And that’s really rewarding. Now, I know not all faculty want more meetings with students. We all have busy schedules. But it’s also great to build those connections, because those are the students who are going to continue to be connected, have mentorship and be successful.

John: And we’ve always been preparing students for their future lives and careers. But we haven’t always been that transparent about it. And it sounds like that’s a really good approach. For someone who wants to start building this into their courses, how would you suggest they get started?

Jessica: I think one of the first things is go have a conversation with someone in your career design center on campus, see what they’re offering and see how you can collaborate with them. I found that our Career Design Center at University of Buffalo has so many resources so that faculty can literally plug in modules on career development that are already created for them. It was also really enlightening to learn what it’s like to be a student, to go to the career design center. And so sitting down with someone and understanding some of the intakes that they do, some of the questions, and even some of the tools that they have, really helps give you an overview of everything that can be offered to students. So when that student comes to you and says, I’m thinking about this career in biostatistics&hellip Oh, great, I don’t actually know a lot about biostatistics, but you should go to the Career Center, because they have a great tool where you can see what your life would be like as a biostatistician. And so first learn about your career design center on campus.. Second, I think it’s important to start slow with any new thing that you’re doing in your course. It may be that you dip your toe in the water and just connect some of what you’re doing with career competencies. So when you have students work in teams, say, you know, teamwork’s important. This is actually what people care about when you get a job. And here’s how you might want to talk about your experience in teamwork, if you don’t currently do this outside of the classroom, and provide them some of those prompts. And then if you want to dive in, and really incorporate your NACE competencies, I think for most professions, in most disciplines, the alignment will be there. And it’s not a ton of effort to highlight that, especially if you start with just your assessments and maybe highlighting some of those and then moving towards other things by maybe adding some career readiness modules, or having your students go to your career design center, take a selfie, whatever they have to offer.

Rebecca: Sometimes it’s really exciting and nice to have some ideas of ways to connect with students that are just so straightforward. [LAUGHTER]

Jessica: I’ll also say, while our students have grown up in a digital age, I find that technology tends to be their weakest category when we think about the NACE competencies. And so if your field uses any technology, talking about how to be proficient in that, how to be a lifelong learner, and how sometimes it’s hard to learn something new, but you have to if you want to keep up into the field,

Rebecca: One of the things that I have been doing in my classes that students have appreciated is when there are things like free online conferences related to the discipline, assigning them as an assignment. And to do that, and to talk about what that experience is like and encourage them to connect with professionals in the field. And a lot more opportunities for that kind of an experience has been offered over the last [LAUGHTER] couple of years.

Jessica: That’s a great way to incorporate that&hellip talks, and even as you introduce a new topic, having them recognize that maybe after this class, you’re not going to be an expert, but here are ways that you can build this skill that you may need. And it might be going out and trying something new, it might be connecting with another center on campus. But recognizing there are resources there for you.

Rebecca: I think underscoring the idea that you need to continue learning in your field is something that students don’t always immediately recognize without us pointing out. They don’t recognize that one of the things they might want to ask in an interview are what the professional development opportunities are, or ways to grow as a professional in their first job.

Jessica: Most definitely, and how they want to do that and how they can identify those areas of growth. That’s something that’s not often transparent. We send them out into the world and say, “Great, you’ve got this degree.” But there may be many areas where they can become a little bit more proficient or dig a little bit deeper into a topic. I want to also highlight that I think that this is so important for our first generation and our URM students. In our undergraduate program, we have about 37% of our students being underrepresented minorities or first-generation college students. And if we don’t talk about this, no one else is going to talk about this. And by becoming someone to turn to about career readiness and about asking those questions of “What do I do for my first interview?” or “How do I prepare for this internship?” &hellipwe’re not going to be able to build that for those students because they might not have someone at home to turn to. As a first-generation college student myself, I found that no one was talking to me about this. And so I think it’s critical in higher education that we think about this as an equity and inclusion component within our curriculum.

Rebecca: That’s a really good point. I’m glad that you underscored that Jessica.

John: And one other thing that I think many people have been doing is bringing in guest speakers&hellip and you can bring in some recent graduates to talk a little bit about some of those pathways. I think we’ve all learned how easy it is to bring people in remotely to give presentations. We don’t have to physically bring them to campuses anymore.

Jessica: Yeah, I think that’s really powerful having career panels. And it’s great to connect with our graduates. I love when they say that what they’ve learned in the class actually helped prepare them for what’s next. We’re not just shooting from the hip here, we really talk to our graduates, understand where we need to focus and continue to improve our program as we continue to grow.

John: We always end with the question: “what’s next?”

Jessica: Ah&hellip summer is upon us. And so it is my time to write up all the papers that I’ve been sitting on over the semester. One of that is the paper on incorporating these competencies into our courses. And I look forward to sharing the results when we get that out there. But I’m really interested in how we work with faculty to help them think about adding career competencies to their courses. Because I don’t think this just has to happen at the undergraduate level. I think the graduate level is also key. I teach both grad and undergrad courses. And as I’m revitalizing our graduate capstone, I really think that this is perfectly aligned, along with our competencies for our accrediting body. And so really, it’s all come together for me, and I’m really excited to see what other folks think about it as the word begins to spread.

Rebecca: Sounds like a really exciting opportunity. And I think you’re right, graduate school is a great place for some of these conversations to be happening.

John: Well, thank you. These are things I think we should all keep in mind, because even if we’re not thinking about career competency as being important for our students, students certainly are. It’s always great talking to you.

Jessica: Thank you very much.

Rebecca: Thanks, Jessica.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

242. Student Podcasts

Student research papers have been ubiquitous in higher education, but there are many ways in which students can demonstrate the skills that they have acquired. In this episode, Megan Remmel joins us to discuss the use of student podcasts as a more engaging alternative to traditional research papers. Megan is an Assistant Professor of Political Science at Bradley University.

Show Notes

Transcript

John: Student research papers have been ubiquitous in higher education, but there are many ways in which students can demonstrate the skills that they have acquired. In this episode, we discuss the use of student podcasts
as a more engaging alternative to traditional research papers.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners. [MUSIC]

Rebecca: I guess today is Megan Remmel, an Assistant Professor of Political Science at Bradley University. Welcome, Megan.

Megan: Hi, thanks for having me.

John: Our teas today are… Megan, are you drinking tea?

Megan: I am not drinking tea. I have rooibos this morning. But I am now currently drinking my coke zero sugar.

John: It’s not that much different than many teas.

Megan: Yes.

Rebecca: Many other rebels join us as well. [LAUGHTER] I have English afternoon today, John.

John: And I have ginger peach black tea, a return to an old favorite as we move towards the end of our semester here.

Rebecca: We’re both getting to things that are comforting.

Megan: Ginger is calming. So, [LAUGHTER] you might need that at the end of the semester.

John: Oh, very much so.

Rebecca: So we invited you here today to discuss a podcast project that you’ve been using in your state and local politics class. Can you tell us a little bit about your project.

Megan: So I was trying to think of a way to make students try to approach things in a more neutral way. Because obviously, in poli sci, there’s a lot of soapbox standing. And I’d previously been using just plain old policy analysis papers. And students don’t love them. And so I was trying to think of a way to get them to do the assignment that I wanted them to do, and having some guidance, because they’ve listened to podcasts before, so they kind of know what some of these are structured like. And so I was hoping that that would help tone down some of the opinionation that can come out of these things. And so I YouTubed, and I found John’s YouTube page, [LAUGHTER] and found his podcast project and contacted him just out of the blue asking him if he had any materials he was willing to share with me. And he did. And those came in very handy in terms of being able to guide students in the project. But it was just me trying to give them a different way to do something. Some of them still opted to do a paper this semester, but I wanted to give them the opportunity to try something different, to maybe be able to say to someone in an interview that they’ve done something in this more kind of digitalformat. So it was trying to open up the possibilities for them in class.

John: And was this a face-to-face class? Or was this an online class or a hybrid class?

Megan: So the first time I tried to do it was last spring, and that was a hybrid class and was admittedly a disaster. But that class was a disaster for numerous reasons, I think hybrid being the prime driver of that. So this class was entirely in person. I did allow them to work in groups if they wanted to, and some of them did, and some of them didn’t. And I had them do two rounds of podcasts. The others who wanted to, wrote a paper and the percentages were equal. And so a number of them who worked in groups the first time around did not work in groups the second time around.[LAUGHTER] So they got to choose their own topics. I gave them a list of I think 10 topics from that section of the course. And so there were restraints, but I let them propose if they wanted to do a topic that was of interest to them. Somehow I managed to have a Sports Communication major in the class, and when we talked about special purpose districts, I mentioned to him that there are stadium districts where cities are basically using taxpayer dollars to do massive overhauls of stadiums. And so that’s where he went. So it was still in political science. It was still state and local politics, but it was something of much greater interest to him personally than say, term limits and state legislatures.

Rebecca: Imagine that.

Megan: I know, shocking, right?

John: So, the students worked in groups, how large were the groups that they work in for these podcasts.

Megan: So, I proposed having them work in twos. This class is a 300-level political science class, but it has historically been required for the history secondary education students and criminal justice students. and poli sci students have gotten more interested in state and local politics, but I don’t think they think it’s as sexy as national level or international politics, so I think that they realize that’s where the jobs are, they’re getting more interested. So the audience is not kind of a typical political science class. Because of that, I have these history secondary education majors, who because of how tight their curriculum is, are in classes all the time and know each other really well and work together and collaborate pretty frequently. And so I did allow one group of three to work together. So I basically increased the requirements. So if they worked in a group of one, they had to have eight peer reviewed sources that they could point to in the script. If they were a group of two, they had to have 12 sources, and then this threesome had to have 16 sources. And then it went from a 10-minute requirement to 15 and a requirement to 20 minutes for that three-person group and the three-person group was actually probably the best podcast I got. And I obviously can’t attribute it to whether it was just the number of them or they’ve also been some of the best students in the class this semester, just generally, so I wasn’t surprised that they did a good job anyway.

John: So you mentioned a script. Did you have students submit a script before they recorded or was that done after the fact?

Megan: So kind of both. I had them pick a topic and then I had them submit either an outline or a script and kind of gave the pros and cons, where an outline is obviously a little more freewheeling and allows for a little more conversational style in the recording, whereas a script would be much more definitive, they wouldn’t be scrambling for words necessarily. So they’d probably have fewer filler words and they could be sure that they weren’t fading off and losing track of what they were saying. So I gave them the option of either, I think the students who wrote scripts just generally did better. So I don’t know if in the future when I do this again, if I’m going to get the option of an outline, or if I’m just going to make them write a script, because those seem to just perform better, but with the script I made them include work cited, and they had to tell me where in the script or where in the outline which source connected to that material. So I was trying to make sure that they were still using peer reviewed sources, they could obviously use stuff from outside of that. But I wanted to make sure they were still using peer reviewed sources, the way that my policy analysis paper kids were. But letting them do it in this less structured style, in comparison to like an eight to 10 page policy analysis paper.

Rebecca: How did students respond to having these options?

Megan: I was a little surprised at how few students wanted to do the paper. [LAUGHTER] I don’t know if they’re just burned out from… I assume the last two years has just been a lot of online writing assignments, for instance, and so they were just scrambling at anything that didn’t involve them having to write in such a structured way. So I have relatively few students select the paper option. So I’d say it was probably three quarters picked the podcast and a quarter picked the paper. And the ones that picked the paper… my pattern deducing… seem to be the seniors in the class. And I think they just wanted to get their paper done and be done and not necessarily have to coordinate with other people. And maybe they have prior experience with less successful group work, for instance, and they were just: “I’m going to trust myself.” But that was kind of the pattern, where the underclassmen were more likely to do the podcast and the ones that seem to be picking the paper were the seniors.

John: Was there any apprehension about recording a podcast? Because when I’ve tried doing this, I know students are often a little bit anxious about things like, “I don’t know if I have the technical skills or have theequipment to do that.”

Megan: Yeah, well, what was great was in the material you sent me, you sent me a lot of options that students could use. So for instance, regardless of what they submitted to me, in terms of the outline, I have them use, I think it was Otter’s transcription. And so they were using some of the sources that you sent to me. And so I think they felt more comfortable. And as they went, I think, obviously, from the first round to the second round, the quality of the recordings went up. And some of them realized that there’s ways on their smartphones to record and that it will partly transcribe for them. So I think they got better as they went, I didn’t try to ding them too much for production value in the rubric. So there is stuff in there just about like, “Please don’t have insane amounts of background noise [LAUGHTER] in your podcast. Maybe don’t record it in your car…” or something like that. So I tried to have a kind of minimum standard, but I wasn’t going to hold it against them if it was kind of fuzzy audio, for instance. But they actually didn’t seem all that apprehensive about the idea. They were better at it than I would have felt.

Rebecca: So the burning question is: “Did they move away from so much opinion and they’re more neutral? Or did they stay pretty opinionated? [LAUGHTER]

Megan: So actually, it went better than I thought it was going to, because spring 2021, when I tried this the first time around, I could not get them out of being on their soapboxes. And when I created the instructions for the policy analysis paper, I frame it as though you were working for a state legislator who knows nothing about the policy topic you’re writing about. And they want a policy brief from you and then recommendations at the end. So, the recommendations part is the “opinion” part. But it’s got to be based in all of the research that you’ve talked about earlier. So if it were about legislative term limits, political science agrees on very little, but this is one thing there’s kind of universal agreement on is that they are bad, and they backfire and do the exact opposite of what we want. So if that’s what the research is finally saying, then the idea is that you would recommend to the state legislator to vote against instituting term limits in the state. So I found that they were generally able to do that… it took the scripts, that initial round, to be like, “some of this language is getting a little feisty,” and “some of this, I’m not seeing any citations behind it, so, as far as I’m concerned, it’s reading like your personal opinion.” So I think that stuff was pretty necessary to get them to tone it down. I also had them submit draft recordings before the final recording. So I could ensure that the script was improved upon for the recording and so I could direct them if they were starting to go a little too far into the opinion editorial page of the newspaper. And so they were generally pretty good at it. If anything, I think they might have been overly cautious by the end of it, in that they had all this evidence about something leading to something and it was kind of repetitive so… confident that that’s actually what’s happening and still feeling like they have to do a both sides-ism. So I think I’m gonna have to try to work on that to instill in them that “No, you can take a position at the end, it’s just got to be based on the evidence you presented earlier, instead of just constantly pontificating,”

John: …and once you have your students do that, could you have them work with some journalists out there? [LAUGHTER]

Megan: Yeah, as somebody who gets interviewed pretty frequently with local media, I get a little frustrated with the both sides-ism. And yesterday, I got interviewed a lot about the Roe draft. And the reporters kept wanting to talk about the leak. And I was like, “No, the leak is not the important part, guys.” So yes, I understand some frustration there.

Rebecca: Can you talk a little bit about how students either shared or heard each other’s podcasts or whether or not the podcasts were shared more broadly.

Megan: So, inside Canvas, which is the learning management software, Bradley uses, for the ones who gave me permission to share, I posted the files inside of Canvas. And then, strangely, and I don’t know if it was because they were maybe afraid of the quality of it, some of them let me share one podcast, but not the other podcasts. And it wasn’t necessarily like, they wouldn’t let me share the first and they would the second, I just think they thought I like this one better, and so you can let people share this one. And I didn’t like this one, and you can’t share this one. So I just put them on the Canvas website. A couple of them told me, the ones who got maybe low Bs, for instance, on the first one, because everyone did pretty well… the ones who got maybe high Cs, low Bs, on the first one, they told me, they went and listened to some of the other podcasts just to kind of see what the universe looked like. And one of them went, “Yeah, I realized I need to step up my game.” [LAUGHTER] And so I think it was useful from that perspective, I don’t necessarily think they were listening to it to learn about the topic that their classmates have done. But I do think it made some of them realize the quality of their work could have been improved If this is the comparison point.

John: I think that’s a useful benefit of any type of peer review of other people’s work, that when they get to see what other people are doing, they might feel better about their own work. But more typically, they realize that there were things they could have done better. And that’s, I think, a useful experience for everyone, including faculty.

Megan: Well, I think students don’t even realize that for all intents and purposes, we have to use them as guinea pigs from semester to semester to make a class better and to improve. So I realized that maybe the pure hybrid format of spring 2021 was not a good time to maybe experiment with assignments. And so it made more sense to try something now. And it’s unfortunate that those kids maybe didn’t get the best experience, but they are our little guinea pigs, and we also need to learn from their work to see how we can make their work better by improving our assignments.

John: One of the issues I’ve had when I’ve done this, I’ve only used it in online classes so far, mostly because my face-to-face classes are relatively large and I couldn’t listen to two or three hundred of these. But one of the issues I had was that for many students in the online classes, during the depths of the pandemic, it was the only time they really got to talk to other students at the same time and I ended up with these incredibly long draft recordings, sometimes, like 30 or 40 minutes for a podcast that was supposed to be quite a bit shorter. And it did add to the amount of time it took to provide feedback. And included in the rubric was a great penalty if it was too short or too long. So I had to remind them of that. It was a tiny penalty, I think the length was only like five or 10% or so of the rubric score, but I felt bad docking them for that, because when I listened to it, it was clear that they were just enjoying getting to know each other and they were having these great conversations and getting to know their classmates. On the other hand, the focus could have been a little bit tighter. And that is one of the trade offs about having a script versus something which is a little more freeform. But it was really encouraging to hear the connections that students were forming. Although, after many hours of this, I would have appreciated them being a little more concise in some of that discussion.

Megan: Well, to your point. I’m curious, I’m not teaching the summer, but I am teaching an online Intro to American Government class this fall. And when I’ve taught it online before I just used forum postings. And it’s a lot of “I agree with this person,” even though you have directions that tell them not to do this, “I agree with what this person said.” And I’m kind of wondering, and thinking about tweaking this for the fall of kind of doing these voice responses, in hopes that it might limit some of that just repetitive nature and get maybe something a little bit more substantive. Plus, it’s more interesting for me than just reading the same post over and over and over again. And because it’s in an online environment, and it’s asynchronous, though I do have weekly benchmarks so they can access everything all at once, I think it would allow them to have a little bit more of the interaction than they get into the standard asynchronous typical shell. So it’s nice to hear that. I think I would also then have to say it was only supposed to be a 300 word post. So that’s only like maybe two paragraphs so we really don’t need to give a War and Peace sort of opinion. But maybe that would give them some of that more conversational style and make them feel like they’re at least possibly getting to know some classmates, ideally with the idea that maybe they can talk to each other and go over course material instead of being in their own little silos.

Rebecca: There’s something about hearing a voice, or seeing a face that can make all the difference. Of course, from your end, if you just make sure they have to post things in accessible format, you can either listen or read, whichever might be faster. [LAUGHTER]

Megan: What are you implying Rebecca? [LAUGHTER] It was more fun to grade these for sure, it was way more fun to grade these than a standard paper, without a doubt, because I could listen and giggle. [LAUGHTER] Because some of them would throw in a little snarky bit. And you don’t get to have that in these really structured formal papers. So, for me, grading wise, it was definitely more enjoyable.

John: For me as well, it was much more fun. And my impression was students had a lot more fun with it than they would have had doing a written assignment.

Megan: Yeah, I still feel like I’m going to continue to allow them to do a written paper. I know, if I had been in their shoes, I probably would have still pursued a written paper. And I had a couple students in there who were just quiet as church mice. And were probably never going to have the self confidence to even attempt a recording. So I think I still need to provide the option, which I did not do last spring. And I think that’s another reason why it might not have been successful is just given the… I mean, you guys know… class personalities vary wildly. And so that class was just very quiet and reserved and not super engaged. And so podcasts probably not the best approach in that class, in hindsight, but I didn’t know that before they got into the class, and I had built the syllabus.

Rebecca: Yeah, those surprises do happen.

Megan: They do, they do. And it’s a little difficult to overhaul your syllabus quite that radically in the middle of the semester.

John: There is something to be said, though, for pushing students a little out of their comfort zone. And in fact, this podcast, in part, got started because of a similar experience that I had, where I was teaching in the Duke Talent Identification Program…

Megan: I remember that.

John: …and they asked me to be on a podcast they had just started. And I said, “Well, I’m really busy, I don’t really have time for this, and I don’t think I’d really be the best person.” So I gave them a list of people’s names who they should contact to be on this. And they said, “Okay, we’ll contact them, too. But we’d like to interview you.” And after trying to get out of it for a while I agreed to do it, [LAUGHTER] and then realized it wasn’t all that bad. And then I came back from Duke that summer and Rebecca and I were talking and I said, “You know I did this podcast and maybe this is something we might want to consider.” And it’s one of the factors that led into this. I wouldn’t have probably had been doing the podcast had I not been pressured a little bit.

Megan: [LAUGHTER] I feel something similar. 18 year old me would not have done the podcast option. 35 year old me who’s done probably 80 media interviews over the last few years…much more competent doing it now.[LAUGHTER]

Rebecca: I think one of the things that I really responded to that you were saying, Megan, is that you were offering options. And that there were two that you did two podcasts or two papers and that if students chose a paper the first time but then heard podcasts, there’s a second thing. So they could do perhaps one of each, right?

Megan: Yeah, so there was more flexibility. I did not have any of them do that. But at least it was a possibility for them. I feel like I don’t know if they misread the syllabus, but it was once I picked a path that is my path, I am locked in for that path. But there was the possibility of it. So maybe some of them in the future will get maybe a little more courageous and go from a paper to a podcast.

Rebecca: Or maybe they go from a podcast to a paper

Megan: …to a paper.

Rebecca: whatever works for them. [LAUGHTER]

Megan: Also true, whatever is most appealing to their preferences.

Rebecca: I really like what you were both saying too, about personalities of students coming out. And that when they might be writing a more traditional paper, it’s just like entire personhood just disappears. And that having that kind of positionality a little bit come out and their personality come out helps us to get to know our students better and to help them get to know each other better when they’re reviewing each other’s work.

Megan: Yeah, there were some students in the class who in class itself were really quiet and then I would hear these little snarky asides in their podcasts and be like, where’s that in class, I want that in class, please give it to me in class.

John: And they would often make connections to their own lives. They were trying to connect their own experiences to what they were learning in class, at least in the podcasts they were doing for me, and those are exactly the type of connections we try to encourage students to make so that they recognize the salience of what they’re studying. I think that was really helpful.

Megan: Yeah, with my history, secondary education students. In the first half of the class, we talked about state-level interest groups. And so I gave them the option to talk about the NEA and the AFT. And most of them picked it because I think they all know that they’re probably future members of one, if not both of those organizations. And I don’t think any of them realized how different those two organizations are, I don’t think they realized not only currently how different they are, but the histories and the motivations behind them are entirely different. And so I think some of them may not join both of those organizations now, [LAUGHTER] when they become teachers, because I don’t think they like the motivations of one group necessarily versus another group. So I do like that maybe this is actually going to impact their workplace environments, and actually how they choose to behave. Same thing with the second half of the course, we talk about tax policy, which I love… shockingly, not of super interest to them… and obviously, property taxes are one of the major sources of education funding for K through 12. And so a lot of them picked that. And they knew maybe that it was bad, I don’t think they realized how bad it was. And at least in theory, some of them seem to have a little fire lit under them. How quickly that the real world maybe extinguishes that is a different story. But at least for now, I think there’s a lot of desire, at least within these particular students, to try to change school funding formulas, for instance. So I actually looked at the roster in advance of the class starting and looked at the majors of the students to try to find topics that were relevant to state and local politics as a political science class, but that students of those majors would actually gravitate toward.

Rebecca: To me that seems like one of the most meaningful choices that you made in your assignment design, because that really hooks a student and keeps them engaged.

Megan: Yeah, forcing them to talk about a topic they do not care about is hugely problematic. I teach our research methods class, and basically, if it’s a quantitative social science paper, it counts. I don’t care what topic it is. And they’re just mind boggled. So one of them, he’s a political science major, but he’s writing his paper on how video games affect stress levels in people. And so they just get to poke around in stuff that they don’t feel like they have permission to poke around in otherwise.

Rebecca: The other thing that I found interesting as a design faculty who does similar things, maybe not a podcast, but we do things that are out in the public, and we might share them, is that I often give models for students to look at that are professional, we might even analyze those together. But it’s not until they see each other’s that all the light bulbs go on. [LAUGHTER] It’s something about seeing a peer get it that all of a sudden helps bring the rest of the students along. And so they’re always clamoring for getting to see each other’s work. And it does improve the overall quality of the work, in my experience overall….

Megan: Yeah.

Rebecca: …despite the fact that they might have these professional models to look at.

Megan: Yeah, I did, because of, again, John’s instructions, I found state and local related podcasts, and linked to some of them. So they could see how they’re talking about policies, but not being super opinionated about them. And I’m looking forward to now that I have permission to share some of these, I teach this class every spring next spring, being able to give them these models of colleagues basically having done this work. So that, yeah, it doesn’t have the same production quality, and there’s no intro music and ad breaks, but they can see that their classmates have managed to do well on this. And they too, can do well on this. I mean, I always provide sample papers, I get permission from students and remove all their identifying information and post those so that students can see like “You can write a research design in my research methods class, it is possible. This was an A, this is what it takes to get an A.” So I’m glad that a few of them gave me permission to share their podcasts. And I think I’m going to share some of the better ones and some of the less better ones so that they can see for themselves, the spectrum of possibility. And if they’re cool with just putting in somewhat minimal effort, then that’s what this podcast sounds like. And if you want to put in the effort that’s gonna get you an A, that’s what this podcast sounded like.

John: One of the things that my students have commented on at the end of the class was that some of them have decided that they really enjoy podcasting, and they started their own or they plan to do one in the future…

Megan: Wow.

John: …and a few of them have also said, “I never listened to podcasts before, but now I’m listening to these podcasts.” So I was really impressed. But it did have these other side effects that I didn’t really anticipate it having.

Megan: I can’t say any of them have told me that but I’d love for at least for them to listen to some more podcasts because clearly, that’s all I listen to in the car. [LAUGHTER] So many podcasts.

Rebecca: I’ve had similar experiences. John, although I haven’t taught a podcast class. I’ve introduced students to podcasts as part of learning materials.

Megan: Yeah.

Rebecca: …and having assignments… and many of them say that they really enjoy that format more than others, but they may have never really experienced it previously.

Megan: Yeah, I can assign them a 10-page article or I can assign them a 30-minute podcast. They definitely like the 30-minute podcast better. They seem to actually listen to it in a way that they don’t with the reading. So yeah, I have been more and more frequently been trying to find either like 5-minute local NPR stories or outright organized podcasts for them to listen to,

John: I’ve been doing more of the same. And I try to find podcasts that have both the audio and a transcript, so that people can choose a modality depending on where they’re working and reading. In some cases, it may be hard to find the time to listen to audio, or they may be constrained in some way and they prefer reading the text. And in other cases, students would much prefer listening to a podcast while they’re walking or exercising, or doing something else. So they have appreciated the choice when it’s used as a basis for discussions or some other assignment.

Megan: I hadn’t even thought of that. But I’m gonna have to think about that for the syllabi for the fall.

Rebecca: The transcripts are really helpful too, because if you are listening, and then you hear the name of something, or you’re not really quite sure how to spell it, or whatever, the transcripts can be really helpful… if those transcripts have been edited, of course.

John: One thing I’ve always been amazed by are the number of people who say they only read the transcript, they never listen to the podcast. Because what I enjoy about it is the narrative and the ability to focus on a conversation while I’m driving or walking or something similar. And reading the transcript would be very much like all the other reading I do, and it just wouldn’t seem as interesting.

Megan: I agree, but to each his own.

Rebecca: There’s a few podcasts that I listen to regularly that I might go back and revisit in transcripts to pull out some notes of things that I wanted to remember. And so I really get very frustrated when podcasts don’t have transcripts for me to do that.

Megan: [LAUGHTER] Fair.

Rebecca: To support my needs.

Megan: This is about me. Thank you very much.

John: It’s important for accessibility purposes as well.

Rebecca: Yeah, definitely. So Megan, you shared a couple of things that you might want to do differently in the future related to this assignment, do you have any other thoughts about how you might frame or structure the assignment a bit differently to continue producing excellent podcasts and your classes?

Megan: I’m curious about the idea of… It’s a 300-level class, and I feel like providing them the topics was a little hand holdy. And so I’m trying to figure out if I should let them pick the topics, because then they have to work a little harder to figure out what might constitute state and local politics. Because I think you can see that in pretty broad ways. But I worry that if I do that, then they’re just going to take some really black and white literal approaches. So I’m not entirely sure what I’m going to do on the topics front of things. And it might just, again, depend on who’s in the class. I think if it were more of a straight up political science class, I’d let them maybe wander a bit more. But since that’s not the audience of this class, typically, I might try to give them a little more structure, since they’re probably of all people gonna go, “I don’t know what you want for me in a class that’s not directly my major.” So that’s thing number one. I am also trying to figure out for the groups, for when they recorded together, given that some of the groups fell apart in the second round, I’ve never had great success with peer review, because they don’t seem to want to be telling the truth about each other. And so I’m trying to figure out how to get around that because it was pretty obvious with one group in particular that someone was doing all of the heavy lifting, but that person wouldn’t fess up to it. And so I’m trying to figure out if I should allow them to work in groups, and if so, how to try to ensure that the workload is being done somewhat more equitably. Because when I’ve had them work on group projects that have involved writing, for instance, I can follow the Google Doc, because I’ve required them to give me access to something like that and I can see who’s adding what. I’m less able to do that in this scenario. So I’m trying to figure out how to… maybe in the script writing process, still have them share it as a Google doc so I can follow the trajectory of who’s adding what, when, in an effort to try to get a better handle on making sure that one person is not being carried through the assignment. And that’s just, I think, a general struggle with trying to grade group work assignments. But that’s the only thing that I can think of right now and I’d love advice. But the only thing that I can think of right now to try to address that a little bit. And I knew it was gonna be a problem, I just didn’t know quite what to do with it. [LAUGHTER]

John: While you can follow the editing history, that can be a bit of a tedious path through that. What I’ve generally asked students to do is just to use a color code where they pick a dark color, so there’s still good visual contrast to meet accessibility issues. But they each have their own color that they write their text in. So when they write a section of a document, just have them block it and choose their color. And then when you read through it, all the dark blue will be from one person, the dark green will be from another person, and the purple will be from a third and it makes it a whole lot easier to evaluate the individual contributions. And that’s worked really well.

Megan: That is beautifully simple. I appreciate that. Thank you.

Rebecca: I do a couple of things, also, because I do a lot of group work, and group work that’s not always visible. I often have shared documents where it makes sense and ways of documenting it. But I’ve also done things like having students keep timesheets, just recording what they’re doing and when and how long it takes with the frame that it might be helpful for them to better understand where they use their time. And so that sometimes is helpful. We also tend to do things in design more like a process document. So it documents the process and milestones and summaries of what they’ve contributed at various points. And so I find those kinds of documents really helpful to understand what people are doing. And I always request them to provide a little bit of information about why they made certain decisions. And as soon as you do that, then you know who did it, it becomes very clear. And the other thing that I do is a Google form as review of the other collaborators. But I do things like ask questions about how willing they were to accept feedback, what was their greatest contribution? So it’s framed in a little bit different way than maybe a traditional rating system? Like, did they do all the things?

Megan: Yeah, it sounds more, what did you do versus what did your partners do?

Rebecca: Yeah. It’s more aligned with how you might do evaluations in a workplace or something rather than maybe traditional peer-to-peer evaluation.

Megan: Those are awesome. Thank you.

Rebecca: Megan, was there anything else that you wanted to make sure we talked about?

Megan: No, I just want to thank John for being so willing to share his materials with me because I would have been floundering about how to start.

John: I was really happy that someone requested it.

Megan: It was great. Thank you so much.

Rebecca: Well, we always wrap up by asking: “What’s next?”

Megan: I think I’ve got a lot of material to work with with state and local politics. Given the recent political landscape, maybe too much material to work with and state local politics. I think one of the things in the future and I’m nowhere near this yet, I’m interested in letting them explore alternate methods of this podcasting style. So maybe actually interviewing local candidates, working with political parties, the League of Women Voters is actually very, very active in Peoria, and they still subscribe to being non-partisan, so it would be nice to try to team up with them and see if they’ve got some sort of outreach campaign they’d like to do. So I’m thinking of trying to really expand, but next year is busy. So that might be the year after.

Rebecca: Well, thank you so much for joining us.

Megan: Thank you for having me.

John: Thank you very much. And I’m really thrilled that someone actually found that material useful. [LAUGHTER]

Megan: It was, it was great. [LAUGHTER]

John: It’s an activity I’ve been doing with my class for three years now, and it’s been working really well, and I’ve really enjoyed it. And many of the students really have to.

Megan: Yeah, time two was the charm for me.

Rebecca: Yeah, you have to have a practice round.

Megan: Yeah, unfortunately, those students were guinea pigs, but I learned from them.

Megan: Thank you for having me. I appreciate it.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Anna Croyle.

[MUSIC]

236. ePortfolios

As David Wiley has noted, “disposable assignments” often have small impacts on student learning. In this episode Nikki Wilson Clasby joins us to  discuss how one campus has used ePortfolios to create authentic learning experiences in their English composition courses.

Nikki is the coordinator of the English Composition Program at SUNY New Paltz.

Show Notes

Transcript

Rebecca: As David Wiley has noted, “disposable assignments” often have small impacts on student learning. In this episode we discuss how one campus has used ePortfolios to create authentic learning experiences in their English composition courses.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Nikki Wilson Clasby. Nikki is the coordinator of the English Composition Program at SUNY New Paltz. Welcome, Nikki.

Nikki: Thank you. It’s good to be here.

John: Thanks for joining us. Our teas today are… Nikki, are you drinking tea?

Nikki: I am certainly drinking tea, yes.

Rebecca: Woohoo!

Nikki: Would you like to know what it is?

Rebecca: Yes!

John: Yes!

Nikki: [LAUGHTER] So this is an exotic blend called Tetley, a very strong British brew, which we Brits love, unless you’re a PG Tips fan, but Tetley’s pretty up there. And I have it with 2% milk which is the best way to drink it.

John: Most of our colleagues on campus from England tend to drink Yorkshire Gold.

Nikki: Mmhmm.

John: They seem to prefer that to the other options.

Nikki: Yes, well I am a Yorkshire lass, but I have to say Tetley has that kick that I need. [LAUGHTER]

Rebecca: Good to know. I think today I have Scottish afternoon tea.

Nikki: Ooh!

John: And continuing with the theme I have an Irish breakfast tea from Twinings.

Rebecca: Oh!

Nikki: Oh very nice, that’s a good one too. I like that one.

Rebecca: This crew needs some strong stuff today.

Nikki: We need some scones now. [LAUGHTER] Then it will be complete.

John: We’re recording this at 12:30 today, and I’ve already had five meetings today including a class.

Rebecca: This is my second pot.

Nikki: I’m impressed. [LAUGHTER] You can come and have tea with me any day.

Rebecca: [LAUGHTER] Perfect. So we’ve invited you here today to discuss the use of ePortfolios in the Composition Program at SUNY New Paltz. But first can you tell us a little bit about your role at New Paltz?

Nikki: Yes, I am the Coordinator of the Composition Program. I stepped into this role two years ago, and I am also a lecturer. And so I teach mainly our upper-level writing and rhetoric courses where I specialize in visual rhetoric. And I also teach courses in what we call Practical Writing and Design which is a new course dealing with a sort of blend of graphics and writing. And I also teach a FIG, a First-year Interest Group, for the Communication Disorders. And I run practicum for our TAs.

Rebecca: So you’re not busy or anything?

Nikki: I’m not busy at all, no. Plenty of time for drinking tea. [LAUGHTER]

John: And plenty of reason to drink that tea with the caffeine.

Nikki: Which is why I drink Tetley, yes. [LAUGHTER]

John: So, we invited you here, though, because we heard about the common use of a WordPress site for the creation of student ePortfolios. And I think the first question we have to ask is… How did you possibly get agreement within a department on the use of one platform?

Nikki: [LAUGHTER] Actually it’s pretty simple, there’s no drama involved here. So in 2019 the composition committee reworked our three-credit English 180 Composition II course to a four-credit GE course, which we retitled English 170 Writing & Rhetoric. And we had been using print portfolios for a long time, and so during the process of revamping our course the composition committee reviewed how we could improve our portfolio assessment. And Matt Newcomb, who was the coordinator at the time, and I had been long advocating for ePortfolios. So during our meetings we decided that it would be a good time, seeing as we were revamping this course, to introduce ePortfolios into our curriculum. And we’d looked at options in Blackboard, but they were just too…

John: Awful? [LAUGHTER]

Nikki: Yes, too awful, but just by happenstance and pure serendipitous coincidence the university at this time decided to just opt for a CampusPress system, and they adopted Hawksites. And so it was made readily available for us to use. And so the timing was perfect, we just jumped on it right away and said, “Yes, this is what we want. This is the way we’re going to go.” So because Matt and I had been advocating for ePortfolios for so long it was pretty simple to get our program on board with the project.

John: And we should note that the Hawks are the campus mascot for SUNY New Paltz. And that Hawksites is just a campus-wide instance of WordPress, I believe.

Nikki: It is, yes. That’s exactly what it is. We just gave it the name Hawksites. Yes, it’s a campus-based university blogging website and ePortfolio tool.

Rebecca: Are students developing these in this more beginning course, and then working on the same portfolio throughout their entire curriculum?

Nikki: Well, this is what’s really interesting. So the faculty are allowed to use the ePortfolios as a tool for however they want to integrate it into their program, so they have free rein to do whatever they want with it. And we all use it in different ways and to different degrees, depending on our comfort level with technology, and how it fits into our curriculum. But as a composition program we use the ePortfolios for assessment purposes, so I can tell you a little bit about how that is organized. So whereas we can have free rein to use them however we want, we do have some very specifics that we need for our assessments. Would you like me to tell you about those?

Rebecca: Love to hear about those.

Nikki: [LAUGHTER] Okay, so for English 160 which is the basic, very first writing course, students have to go through this ePortfolio assessment at the end of the semester to determine whether or not they are fit to move on to English 170. So this is the tool that we use to make sure those students are ready for the more vigorous program. So for that 160 assessment process the students have to upload to the ePortfolio, or at least be able to visibly show on the ePortfolio, they have to have two of their strongest assignments, and they also have to have their revisions for those assignments. They can choose whichever ones they want to put on, but they have to be two major assignments. The only requirement is that they have to show that they have been able to write in different modes, different genres for different rhetorical situations. And there needs to be an element of research in their citation, you know the beginning stages of that research process. And obviously we’ll be looking for the standard of their writing as well, that’s why the revision aspect is really important. So that’s what they’re required to do for the ePortfolio. And then the 170 students, they have different requirements. But let me just backtrack just for one second. So across the board for 160 and 170 as part of the ePortfolio requirements, all the students have to create a reflective cover letter that goes up front in their ePortfolio. They write that reflective cover letter at the end of the semester and we give them questions, guidelines as to what to tackle. And what we want from them is a sort of critical overview of their progress during the course. And they have to cite examples of their writing to prove their case. So it’s a persuasive letter, and our assessors read that first, so they’ll read that reflective cover letter. And that gives us a very clear sense of what the student understands about their writing process, and that makes us feel a little bit better about whether they’re ready for 170 or not. It shows that they’re applying the techniques and skills that they’ve learned throughout the semester to that cover letter. So the 170 students, we have a very specific framework for our 170 Writing & Rhetoric program, it’s based upon a wicked question. So a wicked question might be… What should we eat? Or how do we save the world? Or what does it mean to be human in a digital landscape? And all professors can choose whichever kind of wicked question they want, that they’re excited about, and then they base all of their assignments around that wicked question. That gives us a lot of flexibility for Writing & Rhetoric, which is wonderful. So the semester is divided into two sections. We basically require two major assignments that are argument research based, and then each of those two large assignments has two smaller assignments that help students gear into those big assignments. So, for instance, you might have a proposal with an annotated bibliography that leads to a research paper. So students have to choose one of those sections. So in the ePortfolio we want to see two smaller assignments leading to a large assignment. We don’t need revisions at the stage for 170, we acknowledge that revision is part of the process, and that they will be revising anyway for those papers. So that’s the structure of the assignment sequences for those two ePortfolios. And then beyond that we add other things into the ePortfolios as we see fit. So, for instance, this semester we have our internal assessment which is for our 170 students, and that’s on basic critical reading. So that’s kind of how our ePortfolios are set up. And then at the end of the semester we have a system set up where we review each other’s ePortfolios based on a common rubric that we have put together. So that’s basically how it works.

John: It sounds like a great approach in ensuring standardization across their classes and making sure that all students meet the requirements to move on.

Nikki: It is, it’s very effective. We have lots of conversations afterwards about who’s on the cusp, borderline cases, and so it’s very democratically pieced together. And then of course we have to work on individual cases of students who are failing for various reasons. And it’s a pretty good system, and it’s been very effective over the last two years that we’ve been using it.

John: This is more of a technical question about the organization… Is each site organized by class or is it by students? In other words, does a student have their own WordPress account that they use and create an ePortfolio that is unique to them across all their classes? Or is there a class site where all the students in the class post their work? Or is it some combination of the two?

Nikki: So what happens for us is that in our composition program, we have a template on Hawksites, and our students create an account through Hawksites, and they are given the template that they have to use. So they are essentially creating their own account on Hawksites for our classes, and it’s unique to them. It’s not something that we share with other classes, this is specifically for our class. Does that make sense?

John: Mmhmm.

Rebecca: So just to clarify, if a student was in another class, in another subject area, they may have an additional site.

Nikki: Correct, students can create as many accounts as they want on Hawksites for individual programs. I have about 10. [LAUGHTER] It’s fantastic. It’s such a good resource, we love it.

John: I’m hosting a variety of WordPress sites as well for different purposes.

Rebecca: Me too. One thing I think that’s always important to ask when we’re talking about ePortfolios, is whether or not that student work is public to everyone, public just to members of the community, your classroom community? Or are they private? Or do students have a choice of the privacy settings?

Nikki: The students have a choice of privacy settings. But what we encourage students to do is to select the option that allows only people within our university that have a university ID and login to be able to access it, and only the people that the students give the link to, or the people that the faculty give the link to. And this allows us to share those ePortfolios amongst the people who are going to be assessing. So there is some choice for the students, but it also gives us the option to share easily amongst our colleagues. But I want to add something into this too, that within our template that we had created at Hawksites, we have a permissions policy embedded into the site. And that is a basic form and students can sign it, they can say yes or no. We ask the students, “Would you mind If we shared your portfolio for teaching purposes? Would you mind if we shared some of your work for research or for teaching methods?” And students can pick “yes” or “no” for all of those. And that’s nice to have that there on the ePortfolio, so whenever we’re looking for examples we can check the permissions pledge and see who’s agreed and who’s not agreed, and then of course we respect the students that haven’t signed it. So there are some levels of privacy within our cohort of teaching. There is a blog function on our Hawksite. It’s up to the professor whether they use the blog or not. But as they stand, the students can’t see each other’s ePortfolios, those are private. But there is a blog function within Hawksites, and faculty can choose whether they decide to share that blog function with other students or not. I have used that function for a different project, but I haven’t seen anybody take advantage of that because we also have Blackboard which has its own blog function too.

Rebecca: How have students responded to the idea of using ePortfolios?

Nikki: That’s a really good question. It very much depends on the instructor and how the instructor teaches the ePortfolio component. I can tell you that for TAs who are new at trying to grapple with this technology and pedagogy, some of them have in the past waited till the very end of the semester to have the students upload their work. It’s too stressful for students, they can’t handle it. It’s a lot of work to put a good portfolio together. So I make the TAs have the students sign up for an account within the first two weeks of the semester. And I encourage the TAs to find ways to get the students to engage with their ePortfolio on a low-stakes non-graded level just so that they can learn how to use all of the functions and the tools. And also get them in the habit of using their ePortfolio as a working kind of document, and not something that just gets shoved to the end of the semester. So it really depends on how it’s taught. But if you do teach it with those kinds of sensitivities in mind, and you don’t stress the students out, I find my students in particular love using their ePortfolios. They enjoy engaging with them, they enjoy seeing their work look professional on the site, they enjoy the option of using a more web-based writing process for embedding videos, hyperlinks, uploading images, embedding their beautifully designed Google slide presentations into their site. So they do enjoy that process. I give them time in class to do it so it’s very therapeutic for them. But they also appreciate learning some of the real-life skills that comes with curating an ePortfolio, and they recognize that this will help them later. So the enthusiasm for it is pretty high, and most students feel very proud of their ePortfolios by the end of the semester because they have something to show for all of their hard work, and it looks good. So they’ve adopted it really well. My worry about students is they do all this work, and they hit the submit button for grading, and then that paper disappears down the black hole never to resurface, and then they just move on. And it’s a shame because that work is good work, and we want our students to feel like they have a stake in the writing process, they have a stake in scholarship and research, and the ePortfolios provides a really nice platform for allowing them to think of themselves in that respect, and not just the humble student that struggles, if you like. [LAUGHTER] Helps them feel a little bit more professional.

John: David Wiley refers to those types of assignments that students post in Blackboard, or submit their paper at the end of the term and never see again, as disposable assignments. And having something that looks professional that they have access to, and that they can share and feel good about, is something that students really value. I’ve had students write some books in my class, and they really enjoy seeing this final product. It’s something that they can share with their friends, with their parents, with potential employers, and link to on their resumes and so forth or on LinkedIn, and they’ve appreciated that tremendously. I think you do some of the same, right, Rebecca?

Rebecca: Definitely the students love it when it’s like… It’s a real thing, a real shareable thing, with real audiences. [LAUGHTER]

Nikki: Exactly, and that’s the key thing, right? Especially in rhetoric having that real audience, it’s super, super important.

John: And in my experience it leads to a much higher quality of work when they have that non-disposable assignment. Have you seen the same?

Nikki: Yes, I totally agree. There’s a level of accountability that goes on there. So when you’re racing off an essay at the last minute and submitting it, it disappears into the black hole. With the ePortfolio it comes back to hit you in the face, and you can’t put that stuff on the web, you have to go back and revise it. And it’s really nice being in the classroom and having the students respond to your comments and make those revisions. And then you kind of hear the penny dropping, it’s like, “Oh boy, I really didn’t do this very well, I better snatch this up for the ePortfolio.” And it is very reassuring to see that in action. So yeah, it’s lovely.

John: So how have other composition faculty responded? Are they all comfortable with it? Was there any resistance?

Nikki: That’s a really good question. I know, for me, I’ve been involved in this sort of work for a long time, I came from Iowa State University here and we’ve been working with ePortfolios for a long time. And that switch, going from the paper portfolios which I hated [LAUGHTER] sorry, I hated them… Going from the paper portfolios to the ePortfolio, that’s a big mind switch to go through. So we had to work with our faculty, encourage them to set up a Hawksite of their own so they could experiment, help them feel comfortable sharing those Hawksites in the classroom so they could use that as a teaching tool. So initially there was some learning to do, and that’s great, I mean that’s great, that’s fine, perfect. So it took a while to make that switch to ePortfolios, but now that we’ve made that switch, I think we all recognize that it’s so much more accessible, it’s so much easier to organize, it’s so much easier to assess. We’ve only been doing it since the fall of 2019, but I don’t hear any complaints [LAUGHTER] about the ePortfolio. It is part of what we do now. So it’s good.

John: And that was perfectly timed to be ready for the pandemic.

Nikki: You know, it was perfectly timed for that. And what I like about it, and I think what we all agree we like about it, I encourage the faculty to have the students post the links to the faculty right from the very beginning. So that way we can just go in periodically, and we can just monitor what’s happening on there, and then we can give direct feedback to students about it. So it is, it’s a wonderful tool.

Rebecca: One of the things that you’ve mentioned is this template that you share out, and you mentioned some of the permissions that you allow students to choose. Can you talk about some of the other features of the template itself that you share with students? Like, what are some of the things built into it?

Nikki: Yes, so the template has the tabs already constructed so that students don’t have to work out how to recreate those. Obviously we teach them how to generate new tabs, but the basics are already there. So it has a homepage tab, so we encourage students to post a photograph of themselves and think about how they want to present themselves to a general audience as a student. So they have that, we work on that side of things. And then we just have the tab for the reflective cover page. Then we have the tabs for the individual assignments and their revisions,and then we have the permissions tab. And then we also include on the ePortfolio, this is a new feature, during the pandemic we had a lot of issues with attendance and accommodating students who were sick and who were in quarantine, so what we did was we posted the essentials of the course policies on the ePortfolio. And we had students acknowledge and sign that they had read them, and that they understood what those different policies were for attendance, for assignments, for what they needed to do if they were sick, all of those things we put on their ePortfolio site. So it became a quick reference guide for students that they can just pull it up, and they could see what was required of them. But also for us as faculty when students were suffering, or not keeping on track, or getting to the end of the semester and things were not looking good. We could pull that up, we could see who had signed the pledge and we could say, “Look, policy said that you needed to do this, this, and this, and you didn’t do those things.” And so that helps stem the flow of the great appeals at the end of the semester which I have to deal with. So that worked well for keeping students on track, and keeping that information transparent and clear.

John: And you have it in writing, digital writing.

Nikki: We have it in writing, and the students sign it. So it helps them take accountability for their part in this process. They can’t say, “Oh, I didn’t know about that,” when it’s on the ePortfolio and they’ve signed it. It’s like, “Mm, well apparently at some point you did read this.” So that helps. Not all students read it [LAUGHTER] of course, but at least it’s there though, that’s the important thing. Those documents are not buried somewhere else, they’re visible, they’re right up there. And I think that’s really, really important, and I’m really glad that we decided to do that, especially over the pandemic. It’s been helpful.

John: And I know I always read all the terms and conditions when I sign up for a new software package, and so forth.

Nikki: Of course we do. [LAUGHTER] The other thing that I want to add in there is that, for me, some of the professors do this too, but I have my students create a writing journal tab in their ePortfolio, and they have weekly writing journal prompts in there. And I do that so that students have a safe space just to write, and to reflect on what we’re doing in class, and to apply those ideas to material that they’re interested in. And I set that up because I wanted them to feel like they owned their ePortfolio, that it was their ePortfolio, that it was their personal sort of diary, if you like, of all of their work. So that tab is very important for my classes, and my students enjoy doing that kind of work.

Rebecca: You also mentioned earlier that the work looks professional, so I’m assuming then there’s some stylistic things that are built into the portfolio as well. There’s at least a base look for things, no?

Nikki: There is a very basic look, and I would love to be able to include more design tools in the ePortfolio because we don’t have a choice of font style, we can move our images around [LAUGHTER] to a couple of places. It’s very, very rudimentary, and it would be really lovely if we could add a few more tools in there to make it look even better.

John: So everyone in composition has agreed to use templates, but it sounds like they might use them all differently. Is that correct?

Nikki: Yes. Thankfully, even though the design elements are pretty rudimentary, there are some tools to change the actual overall look within the basic template. The students can change the background image, they can change colors, they can personalize it in a way that suits them which is really nice. So yes, that’s fun, and those are good skills to teach the students as well.

John: What about different sections of the course? Is there a standardization in terms of how the platform is used? Or does that vary from instructor to instructor to some extent?

Nikki: It varies from instructor to instructor depending on their comfortability with technology, and how they want to integrate the ePortfolio into their program. I’m not a standardization sort of person, but we do have… the basic elements for assessment are standardized, they have to have those specific elements for assessment. But apart from that they are free to use those ePortfolios as they wish, and that’s the way that I want it to be.

Rebecca: One of the things that might be helpful for listeners too, earlier you were talking about your assessment process, and that people from other sections review work, that you’re reviewing work of the students of other instructors. So I’m wondering if you’d talk a little bit about the logistics of how that actually works. Because I think for some folks it can be such a big undertaking, so hearing stories of how other people organize those sorts of things can be helpful.

Nikki: Yes, so first of all students have to be eligible for an ePortfolio review, that’s the first step. So students have to have completed all of the assignments and all of their requirements, like the library instruction, oral presentation, all of those things. The student has to have at least a D to be able to be eligible. So that sort of weeds out some of the stuff. And then what the faculty do is we take seven portfolios per class, and that’s a random selection, so you take the first student on your roster, and then every fourth student gets to go in that pile. So each faculty member has seven students randomly selected for ePortfolio assessment. Plus, we have then any student who is borderline, any student that is just clinging on there, or any student that a faculty member is really unsure about, so that goes into the pot too. And then my assistant and I, we create ePortfolio partners and we specifically place, for instance, seasoned faculty members with new TAs. And that’s the way that we do it, so we choose who assesses whose work. And that makes it a very organized system and a fair system, especially for the new TAs who are not sure about what to do, at least they’re working with someone who has experience. So that’s how we do it, and the assessments can take place whenever is convenient for that particular pair, as long as all of the results are all tabulated and submitted by a specific time period. And then after that time period we’ve got some space here to work on ePortfolios that have issues. So once that rudimentary assessment is done then anything anyone is concerned about can be given to my assistant and I, and we’ll go through case by case any of those borderline cases that we’re worried about, we can work on those. So that’s kind of how it works, and it’s a really good system. It works really, really well, it’s very efficient, it’s fair for everybody. At the end of the semester, you know we’re tired, the faculty have already been through all of the ePortfolios and given their verdict, and then we double check with those seven to make sure, it’s really a calibration thing to make sure that everybody’s on the same track. And I need to preface this by saying that all faculty members have to go through a standardized calibration training at two points during the semester, so we make sure that everybody knows how to use the rubric and can apply it effectively. So with those checks and balances it actually works out incredibly well.

John: Are there standard documents that you share with people, and then you see how they evaluated to compare against the benchmarks, for the calibration?

Nikki: Yes, we do. We have a standardized rubric, and then during our retreat sessions we will selectively pick, like, a very, very borderline portfolio for people to assess. We put people in groups, and then we make them grade the ePortfolio with the rubric, and then we discuss it and we talk about what’s working, what’s not working. And if there’s any huge discrepancies in the assessment of those ePortfolios we talk about what was going wrong with those discrepancies. So it’s pretty organized, and it’s pretty efficient.

John: It seems like a really nice way to provide equitable and fair assessments that adhere to the standards that you’re trying to meet. I’m impressed.

Nikki: Thank you.

Rebecca: You talked a little bit about students needing to meet standards to go through the portfolio assessments. Does that essentially equate to their ability to continue on in that particular program?

Nikki: Yes it does. If they’re not meeting the basic requirements for an ePortfolio review, technically it means they’ve failed. And so what we do with those students is we then decide… How did that student fail? Did they fail on their own merit? Or did they fail because they tried and tried and tried but just couldn’t get it? So we have standardized measures here that says, “Okay, so if a student has been trying really hard, and they just didn’t get it, then we will allow that student to repeat the course.” So we have checks and balances there for those students.

John: So this program is used universally in the Composition Program. Have similar practices been adopted by other departments at New Paltz?

Nikki: I’m ashamed to say I don’t know, and the reason for that is because I don’t get out much to see. [LAUGHTER]

John: That’s not uncommon especially during the pandemic.

Nikki: Yeah, I just don’t know, and I feel embarrassed to say that, but I came out of teaching a 4/4 load into this position. So that’ll be one of my next step projects is to figure out who else is using them on campus and talk to them about how they’re using those ePortfolios.

Rebecca: So another thing that is worth considering is… You mentioned that students can choose some privacy settings and things. How long do students have access to these portfolios after they’ve created them?

Nikki: As far as I know students have it for as long as they’re a student.

Rebecca: So we’ve talked about assessment being a primary motivator and maybe some professional skills as being a good motivator for putting ePortfolios in place. But are there other advantages to using student ePortfolios that we should be thinking about?

Nikki: So apart from the ePortfolios for the students being an opportunity to see themselves as professional communicators, to help boost their ethos and their confidence. I think we talked a lot about what the students get from this, but from a faculty’s perspective the ePortfolios are a fantastic tool because they are so accessible, they’re easy to coordinate for assessment, we don’t have to wade through buckets and buckets of paper. And also we don’t have to, [LAUGHTER] I know this sounds like a really minor thing, but when we used to do the paper portfolios we’d do the portfolio assessment, and then we would call the students into our offices to break the news to them whether they’ve passed or not, and give them their paper portfolio back, and a lot of students didn’t come. So we ended up with piles, and piles and piles of portfolios in our offices and it’s like, “Well what do we do with those?” I found it really distressing. You know, if you’ve got four classes and 120 students, and every semester, and then they just pile up, that was distressing. So to switch to the e-system just feels better on my soul, [LAUGHTER] for the planet doing this. But the ePortfolios, they’re just such a good tool for faculty for teaching, for training other faculty, and for sharing what we do with our students with each other, and sharing ideas and seeing what the possibilities are. The ePortfolios just offer so much more potential for pushing what writing and rhetoric is, and what we do with it in the classroom. So from a pedagogical point of view, I can’t imagine going back to paper portfolios. It’s just a fantastically amazing, creative, and soul-satisfying tool to have at your disposal.

John: That’s a really nice, positive note to end on.

Rebecca: So we always wrap up by asking, What’s next?

Nikki: Well, so last summer my colleague Rachel Rigolino and I used Hawksites to develop an online tutor training site because we need more TA tutors in our system to help with the writing program, and that was really successful. And so what we want to do now is to extend that. We would like to develop a Hawksite for our TAs so that we can put all of their innovative teaching ideas into a Hawksite, so that it’s accessible to everybody for sharing ideas. And that’s a really big project. So that’s our next big step, to do that.

Rebecca: Sounds like it’ll be really helpful, and really exciting to work on.

Nikki: I think so. I think it will be vital. [LAUGHTER]

John: This sounds like a really good program, and thank you for joining us, and thank you for sharing this with us.

Nikki: Thank you, it was a pleasure. Thank you so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Anna Croyle, Annalyn Smith, and Joshua Vega.

[MUSIC]

190. Academic Integrity

The global pandemic resulted in a dramatic increase in online instruction. This was accompanied by an expansion of the use of online services that, in return for a fee, provide students with solutions to assignments and exams . In this episode, James M. Pitarresi joins us to discuss strategies that faculty can use to preserve academic integrity in their online courses.

James is a Vice Provost for Online and Innovative Education and the Executive Director of the Center for Learning and Teaching at the State University of New York at Binghamton. He is also a Distinguished Teaching Professor in the Department of Mechanical Engineering at Binghamton.

Shownotes

Transcript

John: The global pandemic resulted in a dramatic increase in online instruction. This was accompanied by an expansion of the use
of online services that, in return for a fee, provide students with solutions to assignments and exams. In this episode, we examine strategies that faculty can use to preserve academic integrity in their online courses.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guest today is James M. Pitarresi. James is a Vice Provost for Online and Innovative Education and the Executive Director of the Center for Learning and Teaching at the State University of New York at Binghamton. He is also a Distinguished Teaching Professor in the Department of Mechanical Engineering at Binghamton. Welcome, James.

James: It’s a pleasure to be here. Thanks for bringing me on board.

Rebecca: Today’s teas are… James, are you drinking tea?

James: I had my cup of tea. I drink Barry’s Irish tea with a little bit of sugar and milk, and I’ve already had it today.

John: And I am drinking Spring Cherry Green tea.

Rebecca: And I’ve got a near neighbor with my Scottish afternoon breakfast.

James: Oh, nice.

Rebecca: …to James, not to John. [LAUGHTER]

John: Yeah, I’m not sure where this one comes from. It’s a Republic of Tea tea.

James: Well, I have to try the Scottish Breakfast. I’ve had the English Breakfast tea.

Rebecca: This one’s a Scottish afternoon, and there’s a morning as well, but I actually prefer the afternoon. It’s a little smoother or something.

James: Ok, Yeah, I’ll look for it.

John: And for many episodes, English afternoon was the preferred which is a little bit harder to find. Well, it’s probably comparable to the Scottish afternoon in terms of ease of locating,

Rebecca: You got to know where to look, John, you got to know where to look.

John: Well, we actually have six packs of twenty in the office where they’ve been sitting since last February when they came in.

Rebecca: We have to hurry up and drink those up.

John: So the global pandemic, which began last March, caused many faculty to shift from face-to-face instruction to online instruction, many for the first time. And we’ve seen a tremendous shift of students from face-to-face to online instruction. And that seems to have been accompanied by a fairly dramatic expansion in the use of online services that facilitate academic dishonesty. And a few years ago, at least on our campus, much of that seemed to be taking place using Course Hero. In the last couple years, much of the expansion seems to have been from Chegg. I saw a statistic recently that between April and August of 2020, the number of student uploads or questions to Chegg approximately doubled, and I think that expansion has continued since then. Why are the sites so popular?

James: Yeah, John, great summary of the challenges we face right now. I’ve done a lot of interviewing with students. And actually I stumbled into this in, I think it was June of 2019, at the American Society of Engineering Education annual conference, I think it was in Tampa. And I was doing one of the keynote breakfast things that are sponsored, McGraw Hill was sponsoring it. And we were talking about the future of educational materials. And the session was well attended by both faculty and students. We had a great panel and I was emceeing it. And the conversation came up about Chegg. Now this is pre-pandemic, and the question came up about Chegg. Many of the faculty hadn’t heard of it. All the students had heard of it. And it was an incredible conversation and eye opener. And what came out of that hour and a half meeting was, for me anyways, “Well, why were students using these type of websites?” …and there are many out there, I think Chegg is probably the leading one right now. And it was fascinating, having the conversation with the students. Of course, fast forward to the pandemic, and this all exploded. But what I’ve discovered, in having both focus groups and individual conversations with students from across the country, is that there seems to be a sense that students don’t want to leave points on the table. So if homework is worth 20%, or 25%, or 50%, they’re going to get all of those points if they can, and most students work hard to try to figure it out. But if they’re stuck, they’re not going to leave those points on the table, they’re going to go someplace to get help. And in the past, that might have been the person living in the dorm down the hall, or maybe you went to tutoring or some other source to get help. But these online sites are a couple mouse clicks away. And so the barrier to entry is so low, that if you’re struggling with a demanding curriculum and other things going on, the temptation is so great. And so what I found, in talking to students using these websites, is one of two general flavors. One was the student that was, “I’m not going to leave points on the table,” and frankly, blurring the line between academic dishonesty and trying to actually learn, and that’s sad. And we’ll probably get into that in a little bit in this conversation. The other group of students were students that were really trying to use these sites to learn, they were stuck on something, they couldn’t get past it. They’d submit, they’d take a look or search around. And to a large extent, they really want to learn. The pandemic came, everybody shifted online, and they took a look around and said, “Well, wait a minute here, this person’s cheating. That person’s cheating. I’m the one not cheating, I’m going to get the low grade because I’m really trying to do well, I’m not a cheat.” I heard this over and over again, “I’m not a cheat, but I know everyone in the class is cheating, and I’m not going to be the one getting a B, when they’re getting an A.” And so it became this crazy dynamic of a mix of “I’m not going to leave points on the table,” and “I’ll be darned everyone else is cheating.” And the barrier to entry is so incredibly low that the students who would never cheat, it’s a mouse click away. Oh, there’s the answer. Yeah, I knew that, hand it in, Sorry, long answer, John, but that sort of summarizes my exposure to it.

John: I actually had a conversation with a student just a few weeks ago that mirrors that exact discussion. Her response when we talked about this is she should have had more faith in her own ability, but she was using it as a crutch because she wasn’t that confident, so she was using this in every one of her courses. And I was just the person who happened to catch her doing it.

Rebecca: Academic dishonesty has been an issue for a very long time. But the pandemic has definitely put a spotlight on, especially this kind of digital version of academic dishonesty, even though that same mouse click was there a year or two ago, and these platforms definitely had traffic, they seem to have increased during the pandemic. Can you talk a little bit about the role of the pandemic in this particular issue?

James: Rebecca, you’re absolutely right. Unfortunately, cheating has been going on for a long time. And, usually, the way most professors adjust to that is, let’s say the homework or something might only be worth 10,15, 20%. It’s not particularly high stakes. It’s designed to be formative anyways. It’s the exams. And so in the past, they were face to face. And while young people tend to be very innovative and brilliant in solutions to circumventing academic honesty, in general, it was very hard to do. The pandemic rolls in. And now, exams are online. So we’re in a situation where the barrier to entry to cheating on an exam when it was face to face was pretty gosh darn high. Now, it’s extremely low, because you’re at home, you’re sitting in your bedroom or whatever, at the kitchen table, you’re taking an online test. Many faculty, many universities don’t use a proctoring service, we can talk a little bit about those, they’re typically a cost added, usually to the student, if they become very expensive to do institutionally. And quite frankly, I have a lot of experience that those systems don’t work, they don’t deter cheating, it’s pretty easy to cheat while these systems are being used. And I’ve unfortunately been involved with faculty that I’ve talked to, from my institution and other institutions where cheating has occurred during the exam while it was being video monitored. So in the arms race of trying to prevent cheating in the online world, we as instructors have a tendency to be a step behind. But you’re right, Rebecca, the pandemic shifted us online, including the big assessments, the big summative assessments, the exams, and we struggled with how to do it. Frankly, it was a little bit of lack of imagination on our part, and maybe an unwillingness or not recognizing that this change was afoot. The faculty that I’ve talked to that have modified their exam processes, have had some success. And interestingly enough faculty who have had upfront discussions with their students about academic honesty, and integrity, and setting standards, and a North star for yourself in terms of what your behavior is, they’ve had success in deterring cheating and academic dishonesty. But yeah, the pandemic brought it on, and it was the shift to online, plain and simple.

John: One of the things I think many teaching centers have been advocating for years is to use more online quizzing that’s automated to take some of the pressure off professors, and also to give students lots of formative assessment, as you’ve suggested, much of which is often done as a low-stakes summative assessment too, where students have multiple attempts. And so many faculty have been routinely creating these large test banks and updating them, but they pretty much all appear online pretty quickly. And the benefits of that, in many classes, have effectively disappeared. What can faculty do, without creating thousands of questions every semester, to get around this issue, to give students the benefits of that low-stakes or no-stakes formative assessment, while still making sure that students are actually learning from and ar not just looking things up on one of the services?

Rebecca: John, are you just looking for personal advice?

James: Chances are, it’s out there, John, I’m sorry to say, but hopefully…

John: I actually have been checking and I have done a few things that make it more difficult. But I’ve also been writing hundreds of new questions every week this semester.

James: Yeah, John, that’s part of the challenge is not only is it a shift in thinking for instructors, but it’s a shift in workload. So one approach is you did the work upfront, you’re writing hundreds of new questions with subtle changes, perhaps. And so it makes it very difficult to keep up with all those. The backside is to have more open-ended problems, but then your grading, so your extra work is on the backend. Part of this is having the conversation with the students, an honest conversation about their learning, connecting what you’re doing in the class, what you’re doing with the subject material to the issues that they’re going to face, perhaps in other courses, perhaps in their life and their career choice… so just an adult conversation, this is why it’s important for you to learn that, and I’ve come up with these low-stakes tests, so you can see where you’re at. And yeah, you can cheat on it. If that’s going to be your approach, then I don’t agree with that, and I think it’s eventually going to get you in trouble. So one approach, John is, yeah, you just do it, you have that conversation, and you say, “Look, it’s like taking your temperature to see if you have a fever. Just go and check. You want to know, if you have a fever, you want to know if you really have this material. I think you got to combine that when you get away from the frequent formative testing… the mantra, frequent formative assessment… when you get away from that, and you’re kind of saying, “Okay, here’s the next level test. This is a summative assessment, I need to know where you’re at with it.” That’s where the challenge is at right now, because the test banks have all been widely distributed. And unless you’re a glutton for punishment, and can write hundreds of questions, it’s going to be a real challenge. We’ve had some luck with using Gradescope. John, Rebecca, have you guys used Gradescope?

John: I have not. I know some colleagues who have, although not on our campus that I’m aware of.

James: We’ve had some, I would say modest success with it. It’s a tool that I think works best in a face-to-face type of exam, but you can do it in an online format. It helps speed up the grading process, it leverages artificial intelligence. There are some technical issues and glitches and so forth. But folks in chemistry and math have been experimenting with it with very good results. And one of the things it does is it can reduce the grading time. And that’s been the big pushback from my colleagues is “I’ll make open-ended problems, but then their grading is so hard.” One thing I did, and again, mechanical engineering, so I’ve got an advantage in that I can ask design-type questions. And what I have done in the past… this is pre-pandemic… is I would assign each student in the class slightly different parameters for a design problem. And then I was able, using some software (I think I used Mathematica or MATLAB, I don’t remember), I was able to run all the different variables and come up with approximately what their solution should look like. And so I split the difference, there was a little bit of upfront work in setting it up and a little bit of extra grading. But here’s the thing, the students loved it. And not because it was a design problem… I would say they like design problems. Here’s the insight. They loved it because they could work together, but they all had their own set of parameters that they had to do on their own. So they were like, “Oh, this is great. We got together, four or five of us got together and talked through it and explained it to each other.” But then everyone had to sit down, and kind of run the numbers for themselves. And they all diverged to slightly different solutions. That was a big insight for me. It would be interesting. I mean, I don’t know how you’d expand that to other disciplines. It would be interesting to go back and try that some more and see if that sort of assessment would get around all this. Now that said, could someone post their specific parameters on Chegg and get the answer? Yeah, unfortunately, the answer might be yes, John.

John: And in fact, I did that with my first econometrics exam when we moved online. I created seven variants of each of seven questions for a class of about 30 students, and nearly all the questions ended up online within about an hour of the test. And the first appearance was within 15 minutes of the test opening. So yeah, they can and they get a custom solution that can then be used by others. And for most of them five or six of the variants ended up appearing online very quickly.

James: Yeah, I wish we had a solution. Because if the three of us had the solution to it, we’d be going up for the initial public offering and starting up our company. But yeah, part of it is student behavior, and helping them understand what’s at stake about their learning. Part of it is changing our behavior as instructors. And while I understand sites like Chegg have introduced, I think it;s called honor shields, and so forth, the colleagues that I’ve talked to said it’s not very effective, they haven’t been happy with it. So yeah, this is a very vexing problem. And one that I don’t see a clear solution to in the future, I will say there was a math professor with a small seminar type class, and he just had oral Zoom exams with each student. He just set up a time and asked them. And the same professor, in his larger class, told students, “I might randomly contact you to explain how you solve a problem on a test.” And I think the fear of that alone probably drove students to study. [LAUGHTER] But yeah, you hate to resort to techniques like that. But there it is.

Rebecca: Especially when the platform’s themselves, like Chegg, they’re well designed for the behavior they want to occur. They’re designed in a way that rewards people uploading content, and so it’s designed in a way to be kind of effective at getting students to upload content because they want content.

James: Yep, exactly.

John: That’s actually the Course Hero model where you get free subscriptions if you upload a certain amount of graded material. And then you get access to materials that other students post. With Chegg, there’s a monthly fee, which I know, because I have to pay it just to keep track of all the cheating that’s taking place in my classes, which is really troubling. But it does have a nice interface, which is fairly similar to the interface that Netflix and similar services use. “If you like this problem, you will also like this problem.” And, in general, you can trace your way through and find many other questions from any given assessment that you posted

Rebecca: …referrals and recommendations. It’s amazing.

James: This is the Amazon model. John, one of the things that I found was interesting in talking to students who use Chegg is it recommends another problem. A number of students said that they enjoyed that, because while they saw the solution and everything, they felt that they were getting more experience with different types of problems, and they like that. But what was really interesting is when I would interview students and talk about their use of Chegg, one of the things that kept coming up, over and over again, is they liked the way, with certain types of problems in Chegg, that there could be hints. There are a whole spectrum of solutions available there. But the ones that are sort of curated, they thought were done very well, every step was explained, there was no “Oh, and it can be shown” and “then completing the algebra you get”… they showed all the steps. And some problems have hints, and you can choose to uncover the hints. And many of the students said they loved that. It was in plain language, and it showed all the steps. And maybe we should take a lesson from that when we put together course materials and study guides for our students, maybe that would be more beneficial for the students, would help them work their way through it. But I thought that was an interesting insight, that those were key aspects. Here’s another aspect, there’s not a lot of video content on Chegg. Now part of it is because “I just need to copy the answer and hand this homework in,” [LAUGHTER] but the students said, “Yeah, they didn’t really care about the video content, they were happy to read through the sort of solution walkthroughs.” And that’s interesting, because that’s sort of the opposite of YouTube, where it’s all video. And so the different learning methods and styles and approaches I thought was interesting. I just wish they didn’t cheat. [LAUGHTER]]

Rebecca: I think taking a lesson from some of those design aspects is important too. A lot of the things that students complain about is the learning management system and what that looks like and feels like or, for example, the problem sets that you’re talking about and wanting it to be in plain language rather than in language that maybe seems too difficult, or it’s not the right level of challenge. All of those things are things that could help the student maybe not want to leave your course and go somewhere else if it was built in. But it all requires a lot of time and resources and materials. And many of us don’t have that available to us with workloads expanding and especially during the pandemic having to turn around things quickly to shift gears,

James: Rebecca, you’re right, the massive shift from face to face to online, the anxiety, especially in the early days over this pandemic, and I don’t think the anxiety has gone down at all. But all that and the extra workload of learning to transition online, and figuring all this out. That’s been a consistent challenge during this whole period, that crazy shift in workload. And it’s been a challenge for all of us. It’s been a challenge for the students, certainly the types of students we’re talking about, in general, are students who went to college, they went to a residential experience. And all of a sudden we told them, [LAUGHTER] “Well, no, you’re actually attending an online institution at this point.” And Rebecca and John, I did want to point out, there was another interesting insight. I was interviewing some students on our campus, and then when I interviewed students from other universities, I was able to find similar things. And here’s what a student said to me, and I won’t use names, but they said, “Oh, in Professor X’s class, we never use Chegg.” Well, why not? “We didn’t have to. Her lectures were great. She explained everything. The homework was tied in, it made sense. And she gave us all her old exams. And she said, ‘Oh, you’re gonna know this for the exam.’” We had her old exams, and we’re like, “Oh my gosh, she’s right. We need to know this for the exam.“ And there was good support and tutorial services. There was great support. This is a quote, one student said “Chegg, Chegg who?” …obviously being a joker about it. But I thought that was fascinating, that when the course is well designed, when the material is presented in a clear way that’s student centered, when the students clearly understand how they’re going to be assessed, they know there isn’t going to be the trick problem to separate the stratospheric A from everybody else, they were like, “Yeah, we don’t need Chegg. We have everything we need from the professor and the student support services.” And when I asked students from other universities, I said, “Well, tell me about a course that you didn’t use Chegg.” And pretty much, I’d say well over half of them had “Oh, yeah, in organic chemistry, in thermodynamics, yeah, I had a great professor.” So there’s something in there that we need to learn as instructors. But that said, another quote from the same students who did use Chegg in another class, when I said “It’s academic dishonesty, you can be expelled.” And he said, “80% of the class is using it, they gonna expel all of us?” That’s an interesting perspective from a student. So yeah, but talking to students and getting their views on this has been tremendously fascinating. And it’s really helped inform a lot of the advice I give my colleagues about how to make the best of this situation. But it’s a challenge.

John: And I suspect the number of classes where it’s not being used has probably declined quite a bit, because once they’re paying that monthly fee, and now Chegg has a really nice mobile app, where you just take a picture of a problem on the screen, it uploads it, and the response comes back generally in 15 to 20 minutes, the marginal cost of engaging it in additional classes has become a lot lower for those students who might have considered it but didn’t think it was worthwhile before.

James: Yeah, the barrier to entry is now at ground level, other than the fee of $15, or whatever it is. It would be nice to engage with Chegg and other platforms. We need the students to learn this material. This is important. Let’s work together. I mean, the honor shield, okay, if it’s not working, why isn’t it working? Let’s figure out why it isn’t working on our end, working with students on academic honesty, and having a North star and having their own internal what’s right and what’s wrong, and helping young people build that set of skills and beliefs about themselves. You mentioned earlier that the young woman, she didn’t have the confidence in herself to be able to just do it. And part of it might be down the road, do departments and schools and universities and so forth, make a big fuss out of this? Is there legal action in the future? This is interesting. Where is this gonna go? I think the article in Forbes, which was very enlightening, they talked about the valuation of the company in the billions. And so if you take a very crass look at this, you got a multi billion dollar company based on cheating. That’s a hard swipe, and so forth. But let’s have a conversation.

John: And it’s basically all copyright infringement of textbook publishers’ content and faculty members’ content. So basically, they’re making millions, essentially from encouraging academic dishonesty and from infringing on everybody’s copyright. So it wouldn’t really be all that difficult, I would think, although copyright law with digital materials is a little bit tricky, because we do have the DMCA out there. And I know Chegg in particular, and I think Course Hero as well, is pretty good at responding to DMCA takedown requests, because I’ve sent dozens of them there just in the past semester, and quite a few over the last couple of years. And Chegg is actually also very good in providing faculty with information on the login ID that students use. Up until last year, in my experience at least, students were mostly using their actual college email address. Now they’ve tended to switch it a bit where they’ve created fake Gmail accounts, but they’re still logging in from the same IP address that they’re using when they submit their exams, which makes it really easy to do a look up between the exam and the person contributing the material. So there are ways of enforcing this. And if more faculty crack down on it, perhaps, it might deter a bit more of this activity. But there’s millions being made, as you said, and it might be nice if some of the publishers would work together to try to take back their ownership of the material they’ve paid to create.

James: I don’t know copyright law. But some of the problems I’ve looked at, where they were explained in more detail, whoever wrote that up, sort of wrote it up independent, they did not photocopy the instructors manual, the solution manual, they worked it out themselves. And truly I do not know what the copyright law is there.

John: The solutions, I think, would not be violations to copyright. But the photocopies of the problems and the test questions and so forth would be a violation of copyright.

James: That’s right, when we make up exams and so forth. But I mean, I’ve talked to faculty who had the exam, and had on the exam, “do not upload to Chegg,” [LAUGHTER] and it appeared, as you said, John, within 15, 20 minutes, and in those cases, it’s pretty easy to prosecute. Chegg will, as long as you go through the official academic honesty policy on campus, they’ll provide information, and that gets ugly really fast for the student. And here’s the other issue, John and Rebecca, I’ve pursued academic dishonesty cases, not involving Chegg, and it is work, it’s effort, it’s stress, and then you get the emails from the student: “You’re ruining my life” and all this kind of stuff. And It can be like, “Oh, boy, wouldn’t have been so much better if we had this conversation and you said, ‘Hey, Professor, I’m really struggling, can you give me some extra help or help me find a tutor.’” Here at Binghamton University, and I assume at many universities, students pay various comprehensive fees. tutoring is free, just go and sign up for tutoring, you’ve already paid a comprehensive fee that covers it. And I make that pitch to the first-year students all the time and say,” You’ve already paid for this, go and use it.” But yeah, I mean, think of the effort it takes for you to then go through all this work and crosswalk an IP address to this and that, I mean, that’s part of the equation we talked about, the time shift, you have to spend time on front creating assessments, on the back end grading it, and all through the process, pursuing it legally, a real challenge.

Rebecca: I think sometimes the argument, too, for students about being honest works a little better when it’s in their major, because there’s a slightly better sell of like a direct impact of “this skill set is really going to get you when you start that job and you can’t do the thing.” But we have to work on our arguments for the courses that might be in general education and things and help students recognize how those are valuable as well out in the workforce. Because I think sometimes that argument can be really compelling for students, but we have to be ready to help make it and help them want to be authentic in what they’re doing so that they can have success.

James: Absolutely. And you’re right. When I talked to students, certainly, for the courses outside of their major, they were much more willing to just survive the course, they really didn’t care. And, again, what lesson can we learn from that? Why are students feeling that way? Why are they saying that? Are those courses not connected? Or did we just not make the connection? We didn’t show them, “Oh, yes, that’s important, that gen ed is really much more important.” I’d argue, certainly in the STEM fields, as so many STEM degrees are being offered worldwide, as technology allows for so many of the things that used to require a person, now can be done by AI, that it’s our ability to work in teams, our ability to communicate, its ethics, and how do you tackle big challenging problems? I might be a mechanical engineer, I bring that background, other people bring other backgrounds and experiences. And that, what a great way to tie in general education courses to the bigger picture. Are we making that argument? Are we helping students make those connections? So, something to think about. I don’t have any answers, guys, so I’ll have to stay tuned to your podcast as you bring smarter people in to say, “Oh, well, when James said that, I have the solution.” [LAUGHTER]

John: Are there some other approaches that faculty could use in place of more traditional exams to eliminate some of the incentives and the possibility for this type of academic integrity concern?

James: Sure, there are, and they’re are more time consuming. So I’m going to be honest with myself, it’s a little easier to have 10 multiple choice questions in mechanical engineering on an exam, a little bit of a testbank, a little bit of taken from the homework, taken from my notes, super easy to grade. If you got 100 plus students, you can grade that pretty quick or you bubble source it. And I justify it, because the licensing exam is multiple choice. Well, it’s okay. It’s okay. I think we got to go back and say, “Well, maybe a judicious blend of some multiple choice questions.” Hey, you know what? The licensing exam is multiple choice. And sometimes you just got to get the right answer. With longer answer prompts that ask them to evaluate something, that’s the approach I use, and it weighs heavily on the back end. The grading is now something that you spend the weekend with a stack of papers,going through them. And you know, you guys know how that is, you can’t start grading a problem and stop halfway because you lose it in your head, right? You got the rubric in front of you. But still, you get a certain “Ok, I took a point off for that, yeah.” And so you got to sit and do that whole problem. And that’s for a lot of folks, I’ll just say, for me, that’s a shift in doing things as we shift to online. I hope that when we go back to face to face, and let’s hope it’s this fall, that we don’t forget some of these lessons, that we really should be designing better assessments that really challenge what the students know. My argument is, if you can google on it, and it’s three mouse clicks away, it’s probably not worthy of testing them on it. And the students are telling you that by saying I’m just gonna copy it. So I don’t have great solutions. I know some of the learning management systems, you can put problems in there, they’ll mix up the order, they’re timed, you can’t go back. But, the only problem I have with that is what is it we’re really testing? The time thing just puts a lot of anxiety and pressure on people and I’m not a good one under those conditions. And you know, John, I’m a “Oh shoot on problem two, wait a minute. Yeah, I did. Oh, I did that. I want to go back to problem two. I just remembered, cause in problem six, there was something similar.” You can’t do it. So yeah, there are things we do out there. But they’re not getting at the heart of what we want. And that is students to learn this material and for us to assess it in a fair and reasonable way, and help the students connect all this for whatever goals they have in their lives. What are you trying to do? Why are you in college? What’s your goal? Let’s connect what we’re doing to that goal. So Chegg’s not in front of that, that’s a deeply philosophical question for another podcast. [LAUGHTER]

Rebecca: I think combined with that, James, depending on your class size, open pedagogy or authentic assessments are also options, but it has to be the right kind of course, with the right kind of content, and the right kind of class size…

James: Yeah.

Rebecca: …for those things to all work in the mix.

James: Yeah, Rebecca, right now I’m teaching an innovation class with 10 students. [LAUGHTER] It is an absolute joy. It’s 10 motivated students, we’re using a platform called mural, which is an online collaborative platform. And so when the homework is due, I can actually go into Mural and see them doing the homework and they see each other’s homework. It’s like, yeah, we’re all going to collaborate on this. And so they can actually look and see what other people are putting in. And I mix it up, I randomize some stuff. So they have to do the reading and then they have to do parts. But, the students love it. And believe me, it’s almost like a bespoke education. This is like hand crafted somewhere between there and I used to teach some of the big sophomore engineering classes with 175 students, and I know some of your listeners who are teaching even bigger classes. That becomes almost industrial in scale. And the ability to give authentic assessments becomes very, very difficult. And until we get those AI engines up… and I’ll tell you what, when they get to the level of being able to do that. we’re all out of a job…. [LAUGHTER] or we better redefine what it is we do as educators.

Rebecca: I think what you’re pointing out, though, James, in some ways is that it tends to be those lower level classes that are bigger, those introductory classes. And so although they might get away with it, in those lower classes, they may not be thinking about the long-term game there.[LAUGHTER]

James: Yeah.

Rebecca: …because those are skills and things that they need in those upper-level classes, especially in their major, it’ll come back to get them in a way that if they don’t take an upper- level class in something else, they might not experience the same kind of consequences. But even reminding students of that, particular long-term consequences of their choices, could be useful.

James: That’s excellent. Rebecca. And as we talked earlier on, I mentioned some of the strategies that seems to work was faculty instructors talking about this, just having a very frank conversation. “Yeah, I know, all these platforms exist. Let’s talk about it. Here’s what I’m trying to do in this class.” Rebecca, I don’t think, in general, we do a great job at connecting the courses across the curriculum. And so I’m teaching my course and I do my thing. And then I hand the students off over the fence. “There you go, go take the next course,” unless we’re doing kind of self studies within our disciplines. “Hey, you know what, I teach this basic thing here at the sophomore level, and you don’t use it till last semester senior year, when does the student practice that thing?” And now we’re expecting them to pull that out of their hat and be experts at it? Why don’t we change things so that they’re constantly using it? And these are great conversations to have… a good hard look, a deep dive into how we build curriculum, helping students connect it, helping connect it to the progress they’re trying to make in their lives. Why are you studying mechanical engineering? What do you want to do with that? Let’s connect those dots through the program, so you can see where things are connected. But as long as the barrier to entry to online cheating is low, [LAUGHTER] but we got an uphill fight.

John: I have tried using some open pedagogy projects, including student-created podcasts and videos created by students, but it’s so much more work evaluating that, that it just doesn’t scale very well. I have been using them in my classes of up to 50 in my online class, but I have not yet committed to doing that in a class of 400, which I normally teach in person… last fall I taught synchronously online.

James: Wow. Yeah, you’re absolutely right. And what are the recurrent themes here? Part of it is: let’s have conversations with students, let’s rethink formative and summative assessments, are there solutions at different scales that make sense? The real challenge, Rebecca, you’re absolutely right, is the big introductory sort of classes where we’ve become used to having sometimes hundreds, if not more, students in the class, these pools of multiple choice questions that make it, and I’m using the phrase industrial and I don’t mean that in a pejorative sense at all, but it’s this ability to be able to expose students to a chemistry curriculum or mechanical engineering curriculum at scale in a very efficient way. [LAUGHTER] I mean, It’s kind of what STEM people do: how do we make this more efficient? And then a big change comes, a huge disruption, and we have to scramble. And so, I don’t see a way around the fact that this is going to be a lot of extra work. And if this were a long term shift, then absolutely, I think we’d be talking about how we restructure higher education in general and different departments and disciplines. I think because it’s short term, the concern I have is, we are going to forget all this, we’re going to wipe our hands of it and go back to face to face and go back to the old way, and not address some of the structural challenges that we’ve uncovered here: helping students understand why it’s important, and “oh, by the way, if the courses are disjointed, that’s on us, go fix that, put some effort into that, and have those conversations with the students.” That’s one of the things I always think about is what is it going to look like a year from now. And I’m concerned that we’re going to miss learning the lessons, we’re going to forget to apply these lessons, when we get back to our old ways, because we’re used to them. “I’ve got my research, my scholarship, my teaching, I’ve got it all balanced just right. I’ve been doing this for a long time, don’t make me change.” But we might have to in order to really help the students be successful. And one thing I always remember is this generation of students is going to be taking care of me when I’m old, so I want them to be good decision makers, [LAUGHTER] and have a good strong set of ethics and a moral compass. So they’re like, “oh, yeah, we have to look at society in a bigger sense.”

Rebecca: That keeps it in perspective, I think, James. [LAUGHTER]

James: Yeah, that’s right. It’s like, “Make sure I’m nice, because when I’m retired, these will be the wage earners. So I want to make sure that they keep the wheels of the economy turning and keep healthcare going and all the things we rely on, that we became painfully aware of during this pandemic, and how thin some of those threads were, how tenuous some of those systems were and are.” So really, it’s amazing how this pandemic has impacted absolutely every aspect of our lives. And we’re talking about some very specific things here. But there’s deep stuff going on here. And this is an opportunity for us to rethink how we move forward.

Rebecca: Especially because it’s easy to go back to what we had or to desire that, especially when we feel potentially burnt out with the workload and things of shifting, or just even having to have the difficult conversations with colleagues about really needing to do significant change when it’s really hard work that needs to be done.

James: Yeah.

Rebecca: So it’s easy to want to avoid it. [LAUGHTER]

James: Yeah, and I’m assuming this is a true statement. But, as a mechanical engineering professor, my training was watching my mechanical engineering professors, right? And their training was watching their professors. And so we teach the way we were taught unless we pause and take a look at the science and research and scholarship in teaching and learning, and then try to apply that. And yeah, Rebecca, you’re right, it’s work. And when you upset that balance, what’s going to happen here? And what does this mean for the future of how we do things? It would be great to not have these massive classes where you could interact with students more directly. I don’t know that that would solve the cheating problem. My students aren’t cheating in my class, there’s only 10 of them, I know them extremely well, they’re very motivated and interested, and they see where it’s connected. So how do you do that at scale and how do you do that across the curriculum?

John: On a more positive note, many faculty who had not been very involved in professional development, who had not reflected on their teaching, because they were just doing it the same way they always had, were suddenly forced to confront some new realities, and they’ve learned a lot during the past year. And I’m hoping that much of what people have learned will not be forgotten as we move past the pandemic.

James: Yeah, I agree. And I’ve been, as I’m sure you’ve seen, I’ve been just blown away by my colleagues, the huge shift, the willingness to just jump on board, to try new technology, to experiment with things and get feedback. Hey, I tried this and it really works great. And then we’ve been able to get that in the hands of other people. It was incredible to see and you’re absolutely right. So let’s pull these good pieces and bring them back. I can see myself now, even when we go face to face, just being more than happy if a student wants to Zoom with me for 15 minutes, and it’s in the evening, I could set up two evenings a week, set up an hour to do that. Like, before, I would have been: “When I get home, I’m toast. I just want to maybe catch up on my reading, catch up on my emails,” I could see that changing. And John and Rebecca, one of the things I have done, is for my entire staff, I have a weekly Zoom town hall where they can ask any questions, I give them updates. It’s fantastic. Folks really like it. And then I run a scholarship program and I just have… every two weeks, I set up a time chunk of time, vast majority of them show up, and we just chat, like, “How are things going?” And I didn’t do that. I mean, I certainly didn’t do it on Zoom before. And I might see individuals now and then, but it made me much more accessible. But in a way that was acceptable. It’s like, “Oh, I’m used to Zoom now, yeah, I don’t mind sitting in my attic studio at home and setting up a 45-minute Zoom and meeting with some students or some colleagues. So what a cool thing to be that accessible and be comfortable with it, within, obviously, within limits, and so forth. That’s a cool thing that I want to continue. A lesson we learned is that when we shifted to online, one thing became apparent right away, is there were a lot of students in socio-economic situations, they didn’t have a laptop, they didn’t have a camera, they didn’t have headphones, they didn’t have internet. And this pandemic has widened the gap to a very uncomfortable level. And so paying attention to that. And what we did at Binghamton is SUNYgave us laptops, we went out and bought a bunch of laptops, we bought mobile MiFi hotspots, 250 of them. You don’t even want to know what my monthly bill is for those… unlimited data. But we just did it. We just did it and sent it out. The other thing is we did a phone campaign just to reach out to students. And we recruited faculty and staff. And it was incredible, just to have a conversation with students. They were like, “Hey,” the overwhelming comment back was “Wow, like, thanks for contacting me.” “Yeah, I’m doing okay,” or “No, I’m not doing okay, I’ve got to take care of my younger brother and sister, my mom is working as a nurse, and it’s absolute chaos here.” So those are two things we learned in the pandemic that I hope we pay attention to, because there are students for with having all this fancy technology, that doesn’t exist. They’re in school, because there’s a computer lab and they don’t own this stuff, they don’t have it. And then the other is just reaching out to students, being a human, you know, “Hey, how’s it going?: …incredible, how powerful that is.

John: This has all become much more visible for faculty as a result of the pandemic. Those inequities were always there, but they were hidden. And now that faculty see that, it may also provide a richer appreciation of the inequities that students face as they were reaching the college level. And that’s something, as we move into the fall, that I think we’re going to see magnified because most students completed most of the last academic year remote, and some students were in well-funded school district with many resources, and all of the students and the faculty had good equipment. And in other schools, they did not have that sort of environment and much less learning occurred. So, we’re going to be faced with a student body that’s going to be experiencing greater inequities as they arrive on our campus in the fall. And I think that’s something that we all have to be prepared for.

James: John, you’re absolutely right, and well said. I think it’s always been there, and now there’s much more awareness. And I think that is something we cannot forget. And the second part of that is the students coming in… we’ve had some conversations with school districts, principals, high school principals, so forth, superintendents, and what we’re hearing is exactly what you said. It’s all over the map in terms of what that academic experience is. We know that the incoming first-year students had a pretty crazy year and a half, a lot of school districts had a “do no harm” policy. So John, if my average in your class was an 87, it didn’t matter. I couldn’t get worse than an 87. So am I motivated to work harder? Or am I going: “and thank you, I just got to take it easy.” And so I may not have learned all that material. So we have that. We have principals telling us that people have moved because of the pandemic, they had to move in with relatives. So the student isn’t showing up, they go to the house, they don’t even live there anymore. Like, where are they? So that’s a challenge. So if you think about the social aspects at home, you think about the emotional growth aspects, you think about the academic aspects, and then you add in standard test optional. The cohort of students coming in is potentially very different in a lot of ways. And we’re going to have to look at how we support them. It’s not a deficit model on their part. It’s a deficit model on our part, too, as instructors. So what are our deficits? And how do we change and modify to meet the students at a place where they can be successful? And I think that’s very important. But John, here’s the thing that was most sobering. One principal told us that it’s not just the seniors, it’s the juniors. It’s like, “Oh, yeah, so not only did the first-year’s propagate through the system, right behind them is the juniors who had a wacky…” and here’s what one principal said: “The biggest failure rate they’re seeing is in the high school sophomores.” This is a problem that isn’t going away. This is going to wash up onto the shores of higher education and if we’re not ready for it, we’re going to be in for a heck of a shock. And quite frankly, those students are going to be in for a shock. So we’ve got to figure out, what do we need to do as institutions of higher learning? And how can we best support students to be successful? No one goes to college to flunk out. They’re going because they’re trying to make progress. They’re trying to make progress in their life. Okay. How do we help them? Oh, yeah, deep and profound stuff, John and Rebecca, the effects of this are going to be with us for years to come.

Rebecca: …at least 13 years, I think, K-12. [LAUGHTER]

James: Oh yeah, right?

John: The preschoolers might have gotten past it by the time they arrive, but…

Rebecca: Yeah. A lot of preschools, I think, still maintained in person, but…

James: Yeah, so big challenges out there. I think we’re up for it. What I love about being in higher education is that we constantly question, we’re curious by nature, stubborn sometimes and have to see data in order to change our minds. But, these are all things we’re good at, and as long as we pay attention, don’t forget the lessons we’ve learned and recognize that the world has changed, and if we’re willing to figure out how we change, then I think this has a good ending. So I’m optimistic. But it’s going to be a lot of work. [LAUGHTER]

Rebecca: Definitely, and some values, of changes of things like flexibility are things that we see as value, perhaps, in students now that we didn’t see before as important skill sets and things, and adaptability.

James: Yeah, and helping students persist, and all the qualities that helped drive us to moving forward in higher education, and so forth. How do we instill some of that curiosity and work ethic? All of us, each one of us, has a unique story of why we went to school and how we want to move forward with our lives. So how do we tap into that and help students be successful? That’s what gets me up in the morning and makes me excited about my job.

Rebecca: And we always wrap up by asking what’s next?

James: Good question. So I can tell you what’s next on my horizon. I’m in the STEM field, I’m doing some research and doing some digging. I’m concerned about traditionally underrepresented groups of students in STEM fields and their success and persistence. I am very fortunate to have an NSF grant that’s sponsoring some work in this area. And when I interview and talk to students, their perspective is very interesting. For example, I was talking with a young Hispanic woman, very smart, already has a job lined up, great, great student. And she said, “Yeah, they bring guest speakers in the class, and no one looks like me, all the guest speakers look like the professor,” …which is like me, an old white guy. So where are the young people? Where are people that, I know it sounds silly, but that look different, but more importantly, have different backgrounds and different experiences and different paths to success. And that’s such an easy thing to fix at all our institutions. We have alumni of diversity who are out there. So I’m concerned about our ability to attract and retain traditionally underrepresented students in STEM fields, because that’s the pipeline for faculty in STEM. So you want to attack the faculty problem in STEM, let’s fix this problem. And you can argue let’s fix K through 12, but I can impact where I’m at right now. And down the road that’s going to impact senior-level administration. So the more people who choose an academic career, the more diverse points of view we have, the more likely that they’ll persist in the career and move into leadership. So, that’s a big problem I’m working on. That’s what’s next for me is some research and scholarship in that area. Because I want the best and brightest students no matter what their background is, because they got to take care of me when I’m old. And I’m already getting pretty old.

Rebecca: I think that’s a perfect note to end on. [LAUGHTER] Thanks so much for that conversation. Really important things to be thinking about.

James: Yeah, good stuff. And thank you so much for inviting me. I really appreciate it. Best wishes to you all. And let’s reconnect in the fall and see what lessons have we hung on to and how crazy is the fall. So, let’s circle back if you don’t mind, I’d love to catch up with you again.

John: That would be great. It’s always great talking to you. Thank you.

James: Great. Awesome. All right. Bye bye.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

183. Student Workload

College students throughout the country have reported substantial increases in their workload during the 2020-21 academic year.  Few faculty members, though, intentionally increased student workloads during this challenging year. In this episode,  Dr. Betsy Barre joins us to explore some reasons for student perceptions of increased workload.

Betsy is the Executive Director of the Center for Advancement of Teaching at Wake Forest University. In 2017 she won with Justin Esarey, the Professional and Organizational Development Network in Higher Education’s Innovation Award for their Course Workload Estimator.

Show Notes

Transcript

John: College students throughout the country have reported substantial increases in their workload during the 2020-21 academic year. Few faculty members, though, intentionally increased student workloads during this challenging year. In this episode, we explore some reasons for student perceptions of increased workload.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist&hellip

John: &hellipand Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Our guest today is Dr. Betsy Barre. Betsy is the Executive Director of the Center for Advancement of Teaching at Wake Forest University. In 2017 she won with Justin Esarey, the Professional and Organizational Development Network in Higher Education’s Innovation Award for their Course Workload Estimator. Welcome back, Betsy.

Betsy: Thanks. It’s great to be back.

John: It’s great to talk to you again. Our teas today are:

Betsy: So, I’m not drinking tea. I’m having many cups of coffee today. [LAUGHTER]

Rebecca: Well, it’s still warm&hellip

Betsy: Yes, that’s right.

Rebecca: &hellipstill warm, and still caffeinated. [LAUGHTER] I’m drinking Irish breakfast today.

John: And I have ginger peach green tea.

Betsy: Nice.

Rebecca: &hellipan old favorite. So we’ve invited here today to talk about your recent blog post that addresses the impact of pandemic instruction on student workload. Can you talk a little bit about how the pandemic has affected student perceptions of their workload?

Betsy: Yeah, sure. So this issue has cropped up for many of us. I’m sure anyone who’s listening to this podcast has&hellip maybe in the spring, but particularly in the fall… and I think that’s really interesting to that in fall it became an even bigger issue than it was last spring&hellip that we started to hear from students in our online courses, and in our blended courses&hellip not just online&hellip that workload was overwhelming, perhaps even double. And we heard it at Wake Forest. We started hearing it anecdotally. And then I would talk to my colleagues at other institutions who, of their own initiative would bring it up, that they had heard it anecdotally as well, we saw on Twitter folks talking about this. And then we at Wake Forest did an all student survey where we didn’t ask about workload&hellip we probably should have. But it was the number one thing that came up in their open ended comments when we coded those. And so it just reinforced this idea that clearly this is a universal challenge. And it was a challenge across our schools to, so it wasn’t just our undergraduate students, We were hearing in our Divinity School and our law school and our business school. And so something was going on. And it was really intriguing to me, because clearly students felt like the workload was overwhelming. But, and this is what we’ve all said. It’s not as if all of us just sat down and said, “We want to give students a lot more work this semester.” So I was fascinated by it, talked to a bunch of people about it, was thinking about it. I know you all have been thinking about it, and just decided to write some of my thoughts in a blog post. One of the great things in the response to that blog post is lots of folks have come up with other ideas that I think are just as plausible too.

Rebecca: Do you think faculty believe that they’re giving more work to their students?

Betsy: That’s actually a really interesting question, because that sort of premise of my blog post is that, and this was Jody Greene said “No one sat down to give more work to students.” But since I’ve written it and talked to some faculty, there are some faculty who are like, “Yeah, maybe I did, maybe I did give a little bit too much work.” And that’s worth noting. But there are just as many faculty, maybe more faculty who say “Actually I have given less work this semester, and I’ve tried to dial it back and lower the stakes than I have in the past.” And so the fact that there’s that large body of faculty that think they’re doing the opposite, and then the student perception is something different. It’s really interesting.

Rebecca: One thing that you just said, Betsy, about the lower stakes piece, raises an interesting question, because a lot of professional development about going online and using effective teaching practices talks a lot about low-stakes assignments and the ability to check in on things more often. But maybe they’re smaller assignments. Do you think that’s happening more?

Betsy: Well, I think it’s a good thing that it is. And my guess is, that’s part of what’s causing the problem or the challenge&hellip maybe it’s not a problem, but just is causing this sort of disconnect… is that our faculty, particularly many of our institutions, in the summer did a lot of professional development around good online teaching practices, and just good teaching practices in general. And also really emphasized&hellip at least at Wake Forest&hellip we really emphasize this is a pandemic, our students are struggling, let’s lower the stakes on things, let’s be understanding. And so one way to do that is by having smaller low-stakes assignments, so instead of a big midterm, you have multiple weekly check-ins. But of course, our students can interpret that as, “more work,” because if you’re just counting work by counting the number of assignments, then it is, by definition, more work. So, instead of one midterm, you now have five short assignments, that’s five times the amount of work. And so instead of counting in terms of how much time the assignments take, they could be counting and just the overwhelming number of assignments seems like more work. And I think that’s what’s going on, or at least part of what’s going on. And I’ve said to some people that this is actually a good sign that change did happen over the summer, because we didn’t hear as much about this in the spring. People kept their one midterm and their final&hellip at least I didn’t hear about it as much, maybe you two did, but I didn’t hear as much about it in the spring. But then they redesigned their courses in the fall. And the fact that we’re all hearing about this suggests that people actually did things differently. Now, again, it still could be better, but that’s kind of a good sign to me. Now, the question is, how do you dial that back? And how do we communicate with students about it&hellip all really complex, but I do think it’s that breaking big assignments into smaller assignments is part of a contributing factor here.

John: . And we know that students tend to do a lot of cramming, they tend to do mass practice, but we know that spaced practice is more helpful and that we know the benefits of retrieval practice. And that’s something I think that most faculty development centers emphasized with faculty. And I know at our campus, we had more faculty participate than we’ve ever seen. We had more people participate in professional development workshops than we generally see over a four- or five-year period. For the people who were resistant to professional development in the past, they were learning about the benefits of retrieval practice and space practiced, and learning about the benefits of using low- stakes exams, as you were just talking about, and I agree that that’s a good thing. But we know that the practices that students use to study tend to be mass practice, they tend to do repeated rereading, and now they’re being asked to retrieve information. And we know that students believe that that’s less effective, and it’s certainly more work for students.

Betsy: So your point about retrieval practice, and we know students believe it’s less effective to be engaging in this continual retrieval practice, I think is really interesting. And I think that’s what we’re seeing when our students say, “We have more busy work.” So it’s not just that there’s more work, but that’s actually more busy work. And part of what’s going on there is that they think that that practice that they’re engaged in is not valuable, if you are giving assignments that are about practice. And as students see it as busy work, that’s part of us communicating the value of this work, and helping our students understand how they actually learn, and how it will help them on the later exams, I think is really important. That’s not the only challenge. I think busy work isn’t the only kind of challenge. It’s also, I think, for those of us in the humanities, I think what we’re seeing is that the new tools we have available to us make it easier for us to hold our students accountable for doing all the reading, when typically, they wouldn’t do all the reading. And typically students wouldn’t say it’s busy work, but there’s more reading that they have to do than they ever had to do before. And so that’s one hypothesis as well. But I think another point about the busy work and the retrieval practice, moving one exam to 10 short assignments is, and I talk about this in the post as well, is that there is a sense in which that could be adding to your work, in that they have to keep track of it all. And I think our students are not used to having to keep track of so many assignments. So typically, as a faculty developer leading a teaching center, I may have a faculty member come to me that wants to redesign their course. And I make all these suggestions, and they do it and it’s fine. And the reason it’s fine is because, yes, it’s a little bit more work and a little bit more stressful for the students. But it’s only one course. But I think what we saw is that all of a sudden, our students were moving from five courses where there were three assignments to five courses where there are 15 to 20 assignments or more. And that was even more compounding the exponential growth that they felt. So I teach with lots of small assignments, I always have. Students would sometimes say this is more work than in a typical class, but they weren’t upset about it. They didn’t feel overwhelmed by it. That’s because again, it wasn’t five of their courses that were doing it. So it is a really interesting question of when we go back post-pandemic, do we want all of our courses to work this way? And how do we help our students readjust to this is the new workload? or this is going to be the new experience of the new workload? Or do we not want to do that? And I think that’s an interesting conversation for all of us to have moving forward.

Rebecca: I know one of the things that I’ve had in conversations with students, just anecdotally, but also in some of the formal research that I’ve been working on related to students with disabilities, is the time management piece and just trying to manage and organize all the moving parts that are on all these different platforms is complex, but also that moving with more materials online has resulted in more reading and writing&hellip

Betsy: interesting.

Rebecca: &helliprather than other modalities that we might typically use in a face-to-face class like face-to-face conversation, which to them seems really much more time consuming. And it may actually be more time consuming, especially if you have a particular kind of disability.

Betsy: Right. So there are a couple of things to say there. So I did say one of the things I noticed is when you read student concerns about this, they will often say things like “It took me this much time to do a discussion post.” And that’s, I think, really revealing for all of us to understand. We often think, “Okay, the discussion posts are going to take the place of the discussion in class.” But right now, I’m just talking to you two, and I’m not thinking very hard about what I’m saying. And in fact, if you created a transcript of this, which you guys probably will, I’ll be embarrassed to read it, because I don’t think it’s as coherent as I want it to be. And if I were writing a discussion post, I would think very carefully about how I formulate my thoughts and my arguments, and even proofread. And it’s gonna take a lot more time, if I’m actually writing it out. And I think that’s really important for us to acknowledge that discussion posts and a discussion are not a one-to-one replacement. Or if we want it to be a one-to-one replacement, then we need to tell our students, we expect you to treat it as if you’re not actually writing something that’s meant to be thoughtful, we just want to hear your opinions about this. So that’s one piece. But then in terms of your point about disabilities, I think it’s really interesting in that all the best practices for Universal Design for Learning, we can revisit, and I didn’t talk about this in my post, but I should have, so thank you, Rebecca, for sharing this wrrinkle. Because I think it’s an important part of it is that giving students options for how they can do this work will also empower them to do things that they think are most efficient for their time. So if they can do a VoiceThread or make a video, or one of the activities that some of our faculty have found very successful as an asynchronous replacement for discussion, is to just put students in groups and tell them, in your own time, you get together, have a Zoom discussion about the material, record it and send it to me ,that you’ve had that discussion. So they actually have a discussion. It’s just sort of asynchronously done. But in general, giving students options, it’s not going to solve every problem, but it does empower them to have choice, because there will be some students who prefer to write than to speak. But there may be something like “I’m tired of writing, I want to actually just speak.” And then in terms of the material, I think there was this recent meta analysis that just came out like last week about video versus text, which was really interesting. And as a humanist, I’m sad to see this, but it’s not surprising that sometimes video can be better for student learning than a text can be. Because I often think, “Oh, I’ll just give them something to read, and that will be the replacement for a lecture.” But maybe sometimes there’s a way in which they’d rather watch somebody talk about that material, rather than read about it.

Rebecca: Or by extension, just listen to the material, like in a podcast or something.

Betsy: Yeah, podcasts are a great opportunity. And we’ve heard students say, when I’m walking around campus, or when I’m working out, and it allows them again, to expand their schedule where they have more time to do things and no screen time, which is something they really appreciate because there’s so much Zoom fatigue, that being able to listen to something where they don’t read online and then have to watch online, they can just listen to it is a real relief for them. Absolutely. Unsurprising you all like podcasts.

Rebecca: Anything that gets us off the screen, actually, is something that I work a lot to do with my design students, because whether it’s a pandemic or not, we spent a lot of time on the screen.

Betsy: Yeah, fair enough. So you’re an expert at this. Yeah. You’ve thought about this. That’s great. Yeah, for sure and I think we need to think about that more.

John: But I know even for people who are teaching asynchronously before, some people have started using new tools. On our campus, for example, people who used to give students readings as a basis for discussions now are having students use hypothesis for the discussions, which means students actually have to open the reading [LAUGHTER] and actually respond to the text, which can take a lot more time than just skimming over the abstract and responding to it. And similarly, I’ve been posting videos for 20 some years in my classes, but now I’m doing it where there’s questions embedded in it, which means they actually have to watch them now for a small portion of their grade. So I think some of the tools that people are using may provide more learning, may provide more engagement, but also is going to take a lot more time than how people use them before. And you noted in your blog post that many students would be able to get by and coast to get the grade they want without doing a lot of the things faculty assume that they did. [LAUGHTER] But again, I’m not sure that’s a bad thing. But it does require more time on average.

Betsy: Yeah, it’s a complicated question. It’s a self report, so it could be even lower than this, but just general self report on how much time students spend each week studying, it’s about 15 hours a week, on average, prior to pandemic, and that is for a full-time student. So imagining 15 credit hours they’re studying, it’s one to one, and many faculty assume or hope that it’s more than one to one. [LAUGHTER] But students are very strategic, they’re learning an important skill and figuring out what does need to get done and what doesn’t need to get done to be able to be successful in a course. And so certainly I appreciate that. But I think recognizing this disconnect is important because it helps us understand why faculty didn’t think they were giving more work, but students actually did have more work because faculty were mistakenly assuming that students were spending 30 hours a week studying when really they were only spending 15. And so being aware of that now helps us have a much more honest conversation about well, what do we expect the standards to be for students, and there are differences across different institutions and different programs. So our graduate professional programs are for folks who are working full time, have different sorts of informal expectations, I think, than others. And so it’s worth it for all of us to come together and to talk about that. But I will say I do think it’s just important to say&hellip I probably said this in the post… but we do know that the more time students spend on a task, the more they will learn. So it’s not just like we’re piling on the hours because we want to punish them or we think that’s just really what rigorous teaching is. It’s that actually we know you’ll learn more if you spend more time thinking about a text or practicing the problems, as you said, John, that this will help you learn more. So you obviously don’t want to expect so much that they can’t do other things they have to do in their life. So that’s the tension. I think my recommendation always be if you have to have a full-time job, you shouldn’t be a full time student, because that’s like too much work. So thinking about how do we calibrate the courses that students taketo how much time they’re actually able to put into it is really important. So yeah, I do think that that’s happening. It’s not the only thing. Again, I also think there are faculty who probably expect too much as well, because we’re not good at estimating how much time it takes for students to do things. I think Hypothesis is a great example. I use Hypothesis in my class, I love it. If you’re a humanist and you haven’t used, or if you have assigned readings and you haven’t used Hypothesis or Perusall go look it up and find it. It’s pretty amazing. But I think that remembering that, yes, it will make them read. So that’s extra time because they’re actually gonna have to read and they’re gonna have to read carefully enough to have good questions. [LAUGHTER] So they can’t skim it, as you said, John, but then all the time it takes to actually read everybody else’s comments, really remembering that and that’s where I as a newbie to online, that was like an aha moment for me when one of my colleagues who’s an expert in online teaching was like, “It’s not just the time it takes for them to write their own discussion posts, it’s also they have to read everybody else’s. There’s extra reading that’s involved.” It’s not just the text itself, but it’s also reading everybody else’s responses and so putting them in groups where they’re responding to fewer people or reading fewer people is a really useful tool. Again, I think probably all of these hypotheses are going on. And it’s worth us being honest about all of them, instead of saying, “Oh, it’s definitely the students,” or “it’s definitely the faculty,” it’s like we’re all in this together, and let’s figure out how we move forward.

John: A nice thing, perhaps, would be to give students information about how much time these tasks take. And it would be nice if there was a tool for that, [LAUGHTER] which I believe that you have created.

Betsy: So yes, we have a tool that actually we made pre-pandemic. But one thing I want to say, because a lot of people have used this tool, and I think sometimes people use it in ways that are asking you to do more than it was intended to do. And that it is very much an estimator. It is not meant to be a calculator, that is the exact amount of time that your students are going to spend on something. And it’s very broad. It was essentially just something that I was interested in creating as I was thinking about how much work I assign students in terms of reading and writing. And the original version of it is very much tilted towards reading and writing. So oftentimes, we hear from STEM folks like “What about problem sets?” And that’s and that’s just the Wild West in terms of how much time students spend on that, it’s much harder to get a handle on it, so it’s not there. But there are places in this estimator where you can add a new assignment that isn’t captured by reading and writing and just give your own estimates for how much time you think students will spend. And the main value of this estimator, I think, is that I found that many of my colleagues, myself included, are just not good at the head math required, we just keep adding these assignments, and we think we have a good sense, but literally sitting down and writing out like “Okay, they have to go to the library to get the source.” Well, it’s gonna take him some time to walk to the library and walk back&hellip like literally things like that, realizing how much time you’re asking your students, and then adding it up can be really valuable. And I would do it sometimes on the back of an envelope, but it was chaos. And so I thought, why can’t we just have a calculator that does that, So we have an old version of the calculator, we have a new version that my colleagues in online education at Wake Forest, Allen Brown, helped us work on to add in discussion posts and video lectures and other things so that it’s a little bit closer to what asynchronous online courses might involve. And it can be a tool for overall assessment, but also individual assignment assessment of like, how much time might it spend for them to do this type of reading or to do these types of videos. And if you disagree with what the estimator says, my favorite feature of the estimator is, you can manually adjust it. So you don’t have to get in arguments with us. Whatever your own assumptions are, you can go in there and put that in, and you’ll still be surprised with what the total amount is probably, at least I often am, that I’m giving more than I realized and I have to go back and make some hard choices. So hopefully, it’s a useful tool for everyone. But as John, you said, one of the best things about it is that allows us to better communicate with our students about what we’re expecting as well. And we’ve heard from so many students who have found it super helpful in the courses that have done this, both students who are struggling, but also students who are crazy overachievers, and who will spend 20 hours on a one-page paper. It’s a real relief to them. Even if they only spend four hours when they’re supposed to spend one, at least it’s four, and not 20. So it helps them manage their time as well.

Rebecca: One of the things that I’ve done, at least on longer term projects, that has worked really well for me and my students is having them keep a timesheet and asking them to divide out tasks. And I pose it to them so that we’re in the design field. So it’s to help them think about how they might price something in the future, so they know how long it takes them. So that’s how you get the buy in. But what it helps me do is see how long it takes them to do certain things. And realize it’s like, “Why did you spend this amount of time doing this thing that was really not important, as other thing was much more important?” And then you can coach the group on those sorts of things, which can be helpful. And along those same lines, one of the things that I run into, and this may fit more into the idea of problem sets or things like this is how much time students will try to problem solve a technical issue that they just aren’t problem solving in the right way at all. And so they could spend hours trying to do something that if they just asked a question… [LAUGHTER] &hellipit would have taken two minutes.

Betsy: Like ask for directions…. [LAUGHTER]

Rebecca: Yeah, so I’ve been reminding my students, especially since the fall, when we’ve been doing much more online,that, if you’re spending more than 15 minutes trying to solve this technical problem, A. take a break, you’re just going in circles, maybe come back and try again. But if you’re spending much more time than that, then that’s a good clue that you need to ask for help.

Betsy: That’s really smart. And really, I think, super helpful. And I think getting feedback from our students about how much time they’re spending is not just good at the individual level of coaching. It also is great formative feedback for adjusting our own expectations. Again, and it corrects the estimator, maybe you put it in the estimator, and has happened to me too. And I realized&hellip because one of the things about the estimator, it’s best about reading, usually, in terms of its reading estimates, but one of the central insights from the reading literature is that the difficulty of a text is just as much about students’ vocabulary as it is about the text itself. So I would guess “this is a pretty easy text for my second year students at Wake Forest.” And then if they’re all taking a lot longer, what I realized is that actually, I misjudged their familiarity with these concepts that would be in this book. That this book is actually harder than I thought it would be. So I need to up it in terms of the estimator to say “Actually, there are more new concepts than I realized that the students are engaging with and it’s going to take more time.” So asking the students is just as important as you communicating with them. It’s a two-way street for sure to get that formative feedback. I also think telling them about time management and struggling with time management. I’ve seen some really good strategies. I know our learning assistant center, who works with students, has some good counseling that they do with students about how do they create a master syllabus or kind of a calendar for when they’re going to do things. And I also saw somebody, I think, shared it on the POD listserv, but a strategy of creating a Google calendar with basically time slots for all of your activities in your course. And then students import it into their Google Calendar and move those around. So you would set it up like two hours for reading this text. And then they could move it in their calendar. And so that works for them. But they basically see the blocks of time that they need to set aside. And if they did that for every class, it would be even better, they could see “Oh, wow, this is 40 hours in a week, I need to set aside time to do this work.” And frankly, we should be doing that even before the pandemic. But we’re learning this lesson now of how to help our students manage time and due dates, and all of that, because it is a little bit more. And again, I also want to emphasize too, not just all the cognitive load of multiple assignments, but learning new tools also takes time. This is kind of your point about troubleshooting, Rebecca, like, if a student has never used the video function on Canvas, they may find themselves spending 45 minutes trying to get the video function to work, when that’s not in any of our calculations of their assignment. We’re assuming they’re just going to record the video and upload it. So being mindful of the time it takes them to learn a new tool in this scenario is also really important.

John: You mentioned the issue of reading tied to students prior knowledge and vocabulary. But that’s going to vary a lot across students. So I know a lot of people, when they include estimates from the calculator, will say this is an estimate of what this is, your mileage may vary and keep track of how long it takes you to do these things, and use that to adjust your future estimates of the time requirements for these tasks.

Betsy: That’s a nice idea too, to say you students adjust. So that’s really smart. I like that a lot. For sure, it varies across students. And especially, I mean, even thinking about students with disabilities is an even more interesting challenge. And there is an interesting question, I’ve had some good conversations about to what extent, if we’re putting that estimate… the average&hellip in the syllabus does that create problems for students who may be slower, they think that there’s a deficit. So you need to be thinking about how you frame it, I think is really important. And to be up front that saying it is expected and that is the normal course of things that we’ll all have different rates and this is a ballpark average. You can even put a range&hellip might be an idea too&hellip of ballparks there, but recognizing and saying it’s totally understandable that there’ll be jeans taking a different amount of time, because again, prior knowledge, not just ability, it’s all sorts of other things. How often have you read in the past? How often have you worked with technology in the past? Any of these things, they’re gonna make a difference.

Rebecca: One of the things that conversations about perceptions of workload lead me to is I wonder what the perceptions of learning are?

Betsy: Yeah, I think this is a great question. Because when we think about how students got “got by” in the past by doing less work, what they meant by “get by” was successfully complete the course and get the grade that they desired. If we actually ask them about how much they learn, I don’t know. I mean, that’s a really interesting question, would they say, “Oh, well, it takes more to get my A now&hellip” so that’s duplicating the workload&hellip “But, oh, by the way, I’m also learning more.” It’d be interesting to see. I mean, it depends on f the primary issue here is that students doing less work before and now they’re doing all the work we expected of them, then I think you would expect a lot more learning. But there could also be these issues of the pandemic, I’m in crisis, I can’t work as quickly. If those are the issues, or I’m overwhelmed by the multiple assignments, and I can’t keep track, then there may not be as much learning happening. So my guess is there’s probably equal levels of learning, it’s totally a guess. But in other words, that there are challenges to this moment that students learn less. But there are also things that we’re doing better than we have in the past that make up for that. But I hope that we get some good empirical data on some of this and think through it, because I do think that these strategies, while they are more work, are also probably likely to lead to deeper and lasting learning as well, if the students are able to do it. There’s also the challenge of students who just give up, and then get overwhelmed, and they’re just completely behind. And then they have no motivation to even do a little bit. And so we want to be mindful of that too. But if they’re able to keep up, I’m hopeful at least, that these things should, at least from the research, they should lead to more learning, but who knows.

John: In terms of student reaction, though, student perceptions of what’s most effective is often passive learning and repeated reading. Fluency illusion makes it seem that you’ve mastered the material without being confronted with some type of evidence that you really don’t know this stuff quite as well. And that all the techniques that we’re actively encouraging in teaching centers are giving students more feedback more regularly about what they know, and what they don’t know. And that doesn’t feel as good. And there was a study at Harvard about a year and a half or so ago, where they surveyed students on how they perceived their learning, relative to the actual learning gains they receive across both lecture-based classes and classes that relied on active learning. And there have been a lot of such studies where in general, the students believe that active learning is not as effective yet the learning gains tend to be significantly greater. So there is a bit of a disconnect between what students perceive as being effective and what actually is effective, which also can lead to that perception of busy work that you mentioned before?

Betsy: Yeah, I absolutely agree with that. I think, and this would be a whole other podcast you probably all have done all these podcasts thinking about this issue of student perceptions about learning. I think part of it is what they’re used to. There’s a lot of things that are going into helping all of us understand how we learn and what works and what doesn’t work. And so I think there is a hope for us to try to sort of bring them along with us, I guess I would say, I guess the valuable insight from the studies is that we shouldn’t take for granted that if students say they’re not learning, and if they aren’t learning, that we need to recognize that they may be. And so part of our job is to help them understand, with hopefully concrete evidence that we can show them, “Look, you’re actually learning here in significant ways” &hellipto help them understand why we are choosing these approaches. And it’s not just because we don’t want to teach or we’re lazy, or what are other stories people tell about active learning, moving forward. So I think part of the way we bring them along is to also acknowledge that sometimes there may be assignments that are not useful, and that there may be sometimes things that are overly burdensome in terms of time. And so instead of just always being “You’re wrong, students, you’re wrong here, let us tell you how it is,” to say, “Okay, let’s listen to our students” and say, “Actually, that assignment, it took more time than it was worth. And so we’re going to think creatively together about things that will work for you.” But also acknowledging that there’s a long literature on how people learn that should inform it, and not just perceptions that make a difference.

Rebecca: I think when I’ve even asked students about some of those things like “What do you wish you had more of?” &hellipthey do realize that when you have those little assignments to hold them accountable, and help them practice, I had students asking for more. In the fall, I had students saying “We had a few of those, those were helpful. We wish we had more of those.”

Betsy: Yeah, there’s no question. We saw that in our survey, too. And that’s the reality of anytime you do a study, it’s an average. On average, students think they’re learning less, but they’re always going to be students who, “Oh, I’m aware, I’ve seen this happening.” And they’ll be students who sort of totally missed the boat. But yeah, we saw that for sure. We saw students who appreciated the check-ins, but the number one thing that we saw from our students on various questions was that they wanted more opportunities to work with each other, which, normally, they don’t like that. And there’s literature, right? It’s like, “Oh, I want to be taught by a teacher and not my peer.” But in the pandemic moment when they don’t get to connect with their peers, like socially. So our students are back on campus at Wake Forest. But there’s lots of restrictions on what they’re allowed to do with each other socially. So especially for some of our first-year students who hadn’t made friends yet, this was their opportunity. Classroom collaboration was their opportunity to make friends. And so yes, it was tied to their learning. But they also really just appreciated it and said, “I want to be able to work more, they helped me understand the material more.” So they were calling out both the sort of friendship aspect, the social aspect, and saying, “Oh, it helped me feel more confident in the material, because I could ask questions.” So I certainly think it’s not a universal story, that students are upset about these kinds of active learning and small stakes things. But it’s more universal. I think that they feel like there’s a lot more work. And so that’s what’s so interesting. Rarely do you have a finding or experience where so many people are in an agreement about this. And so it’s just such an interesting thing that I have not met a person who said, “I felt like I had less work.” That’s kind of interesting. But there was one student in our survey, I think I quoted this in our blog post, that was really interesting, where she said, “The courses are easier, but they’re emotionally more difficult.” So the online courses are easier, but it’s emotionally more difficult and more difficult to try hard for. One of my hypotheses was that being in a pandemic makes our capacity to work lower. And so I think that’s part of what that person was getting at. Everything feels like more work, even if it’s the same amount of work. And I am guessing that it’s both that and also maybe a little bit more work too, that’s going on, I’m going to be curious to see what happens in the spring. We’re gonna do our survey again. And we did have some interventions where we talked about this, but there’s no mandates about what people are going to do. So we’re going to ask our faculty, again, what they’re doing. And then we’re going to ask our students and see if things got better. And hopefully, that’ll make us understand maybe which hypotheses are more or less likely to be true? Who knows?

Rebecca: If anything, at least, this is something faculty and students all have in common. We all feel like we have more work.

Betsy: Yeah, well, [LAUGHTER] and actually, we didn’t even mention this. And I didn’t mention in my blog post, because it was already too long, is some of this switch to low-stakes assignments also increases the workload for faculty. You don’t have to assess it all, but many of us are just used to that, so we look at everything and grade everything. And so certainly, we heard a lot on our faculty survey of “I cannot sustain this for another semester.” So, this semester, we may find that many of them have shifted back to fewer, larger assignments. So I’m not sure. We just heard some anecdotes, but I could see that happening too, for their own workload sake as well.

John: In addition to the trauma of the pandemic and all the issues associated with that, I believe you also mentioned the fact that many students signed up for face-to-face classes and just being in an online environment is going to make them less happy. And if you’re not as happy in that environment, it’s going to seem like more work.

Betsy: That’s right. Yeah. And this is where I had a throwback to my own time tracking that I did. Maybe five or six years ago, I did time tracking of my own time and I was fascinated because I wasn’t very good at predicting what I was spending my time on. If I didn’t like being in a meeting, I felt like it dragged on and on and on. If I was reading a book that was really exciting. I thought it was like this [finger snap]. But actually, if I went back, “Oh, I was actually spending a lot of time” or even just working on a design project, I would just lose hours staying up till however many hours in the night because it’s exciting to me, it doesn’t feel like work. And so my guess is that there’s some of that going on, too. And I will say in our survey, there was a group of students who were really unhappy with online learning in general, not specific teachers, not specific strategies&hellip that they did not want online. And so those students, obviously, if they had that much anger and sadness about being online, I can’t imagine that they would be excited and enjoying&hellip like, just another 15 minutes of online would be a slog for them, you know, and so I’m sure that things are slower, because they’re not enjoying it, because they didn’t choose it. And I think that’s a really important thing for all of us in higher ed to be thinking about is that, just because there are some students who are unhappy with online right now doesn’t mean that online itself is the problem. It’s partially giving our students autonomy and choice of how they experience their courses. And there are some of our students who just really want to be in person. And those students are probably also the people who really want to be socializing with their friends. And they aren’t getting any of that right now. And so they’re doubly upset, triply upset, like many of us, and that’s not a good position to be in to enjoy your work&hellip the work is really work. I’m sure that some of that’s going on.

John: I spent a decade working on our faculty assembly one semester. [LAUGHTER]

Betsy: I like that. Yeah, there you go. That’s true, right. Sometimes there’s something that just drags on. Time is tricky like that. Some of our students also commented on just sitting in their dorm rooms all day on their computer screen all day, and leaving to get takeout food and coming back. And they’re in singles, often&hellip a lot of them are in singles, because we de-densified our dorms, like it’s just not a great mode of existence. And so anything they can do to get away from the screen, as Rebecca, as you said, that I think is a really valuable strategy for all of us to try to incorporate into our courses.

Rebecca: I’ve noticed this semester, in my classes, I have really good engagement. They’re synchronous online, I can see people contributing. But there’s a lot less camera use this semester than there was even last semester with some of the same students. And maybe it’s the winter slog, “Oh, the winter won’t end.” But it’s just also just being on screen and feeling almost like you’re in performing mode. I think it’s some of that, too. I’d like to turn my screen off sometimes.

Betsy: I was just gonna say that for those of us that are in committee meetings all the time with our colleagues, like we’re still with screens on all day. And yeah, I absolutely think that there’s just an exhaustion and awareness that there’s another semester of this, we don’t know when it’s going to end. &hellipreally tough, certainly.

Rebecca: So we always wrap up by asking what’s next?

Betsy: Well, if I knew what was next for the fall, I’d be a millionaire right now. Who knows what’s next for the fall? I think that’s the biggest challenge for all of us, as we’re thinking about higher ed, in the near term, at least is what’s going to happen in the fall. But I do think with respect to the topic of this podcast, we often talk about when the pandemic ends. It’s going to be like a trickle, I think. There’s not going to be a sharp ending to it. But whenever we start talking about the future of higher ed in a serious way, I do think there’s going to be a very interesting question about how much do we expect of our students outside of class? And what is an appropriate workload? What is the nature of a credit hour? All of those kinds of questions should be on the table because I know for a fact that many of my faculty, even when they go back to in-person are going to want to keep using the strategies. They’ve read the research that we presented to them this summer, and they see that it’s valuable and that their students are learning and so it’s not as if the workload is going to decrease dramatically, I’m guessing, when we go back to in person, so we may need to have larger conversations about that in higher ed.

Rebecca: Thanks so much for joining us, Betsy. It’s always a pleasure.

Betsy: It was great to be back. I love this podcast. Thanks so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

157. Takeover

Tea for Teaching has been taken over this week by a couple of our favorite authors! Join our friends, Sarah Rose Cavanagh and Josh Eyler, as they interview each other about their current book projects.

Sarah is the author of The Spark of Learning: Energizing Education with the Science of Emotion and of Hivemind: Thinking Alike in a Divided World and numerous scholarly publications. She is the Associate Director for Grants and Research at the D’Amour Center for Teaching Excellence at Assumption College, the Co-Director of the Laboratory for Cognitive and Affective Science, and also Research Affiliate at the Emotion, Brain, and Behavior Laboratory at Tufts University. Josh is the Director of Faculty Development and a lecturer in Writing and Rhetoric at the University of Mississippi. Josh is the author of How Humans Learn: The Science and Stories behind Effective Teaching.

Show Notes

  • The tweet containing the graph discussed in the podcast.
  • Eyler, J. R. (2018). How Humans Learn: The Science and Stories behind Effective College Teaching. West Virginia University Press.
  • Freire, P. (1970). Pedagogy of the oppressed, Herder & Herder.
  • Hooks, B. (2014). Teaching to transgress. Routledge.
  • Cavanagh, Sarah Rose (2019). The Best (and Worst) Ways to Respond to Student Anxiety. The Chronicle of Higher Education. May 5.
  • Yeager, D., Walton, G., & Cohen, G. L. (2013). Addressing achievement gaps with psychological interventions. Phi Delta Kappan, 94(5), 62-65.
  • Cavanagh, S. R. (2016). The Spark of Learning: Energizing the College Classroom with the Science of Emotion. West Virginia University Press.
  • Cavanagh, S. R. (2019). Hivemind: The New Science of Tribalism in our Divided World. Grand Central Publishing.

Transcript

John: Tea for Teaching has been taken over this week by a couple of our favorite authors! Join our friends, Sarah and Josh, as they interview each other about their current book projects.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Sarah: Hello, Josh.

Josh: Hello, Sarah.

Sarah: So, I’m Sarah Rose Cavanagh. I work at Assumption University, where I am an Associate Professor of Psychology and also Associate Director of the D’Amour Center for Teaching Excellence. And since we’re here on Tea for Teaching, I will tell you that I am drinking water, because I have had an almost toxic amount of coffee already today.

Josh: And I am Josh Eyler, the Director of Faculty Development and Faculty in the Department of Writing and Rhetoric at the University of Mississippi. And I always feel as if I disappoint John and Rebecca on this podcast, because I’m drinking water as well. I never drink tea. [LAUGHTER] And we’re here for, I won’t say a special episode of the podcast, but it is a different episode, and we’re hoping not to crash it. So Sarah, do you want to tell everyone what we’re doing here?

Sarah: Absolutely. So, we both announced on Twitter that we have new book projects. And I wanted to interview Josh for his new book project. And he said maybe we could interview each other. And then John said, “Hey, you want a space to do that, publicly?” And so here we are interviewing each other about our new book projects on Tea for Teaching. We’ve taken over the podcast.

Josh: Yes, we have. And it’s my go round to ask the first question. But, I just want to say first, it’s always a lot of fun to talk with Sarah about her work, and so it’s a special honor today to hear about her new work. I’m working on two projects, a book on grades, which will be making up the bulk of the questions, but a smaller project on test anxiety and other kinds of academic anxieties. And so I have a question about that as well. And that’s the one that I’d actually like to start with. And so, Sarah, I’d like to ask you about the attributes of class climate that can lead to anxiousness. I know that you’re working on a project that has to do with our learning environments, and its connections to student wellbeing. And so I’m curious on your take, both from your own perspective, and from the research that you’re doing. What are some of the elements of class climates that can lead to anxiousness in students?

Sarah: Awesome, and thank you for that question. I think that our projects are very related, which isn’t surprising. We also are working together on a grant project that, in part, tackles the question of emotions and grading. And so, for me, anxiety is all about uncertainty. And when we’re unsure of our footing, when we think “what if this terrible thing happens?” when we can’t picture the future? these are conditions under which anxiety arises. So, in terms of classroom climate, the sorts of things that research suggests make students more anxious have to do with uncertainty and with a lack of clarity. And so when students don’t understand the material, when they feel like the material is presented in such a way that they cannot understand it, when there’s clarity and uncertainty there, that makes them feel anxious. When they’re not quite sure where they should be investing their time, and what they should be doing with their time and how their grade is going to be determined, that can also lead to anxiety. And when they are unsure of how they’ll be assessed, what a good result looks like, what the instructor is looking for… and every instructor grades differently, assesses differently, is looking for slightly different things. And I always felt as a student, that part of beginning a class was figuring out what does this instructor want? And what are they looking for? And so all of these things can increase anxiety. I think that climates that reduce anxiety are ones that help students feel safe, help them feel like they belong, and ones that are really transparent and clear. And one of the things I’m trying to tackle in my new project is that we need students to be able to grapple with uncertainty. And I think that we need learning environments in which students feel safe, and they feel like they belong, so that they can tackle uncertainty and so that they can be a little unsteady and a little unsure, without anxiety spiking.

Josh: Great, thank you so much.

Sarah: Alright. So you recently shared on Twitter a graph, and I’m going to try to describe it, [LAUGHTER] because we’re all audio, but it was about grading alternatives. And you had two axes crossing each other. So, if the audience can picture four different quadrants, one of the axes was liberation, from not liberating at all to very liberating… a grading scheme. And then the other axis had to do with logistics, fewer to more logistics. So, various grading schemes were in the different quadrants. We can think of specifications or mastery grading, which have lots of logistics. We can think of ungrading, which has fewer logistics. So, I have a couple questions about that. But, my first question is, what is your operational definition of liberation or liberating? What are its characteristics? Its defining attributes? How do we measure it? This is a psychology question. [LAUGHTER] What is ”liberation”?

Josh: Right, definitely. And thank you for that question. I should say that this book is kind of a first step in a new development for the way I’m taking my writing. It’s different from the last book, in that it doesn’t focus solely on higher ed, but looks at educational systems more broadly, from preschool to grad school, it’s the way I’ve kind of been describing it. And so that’s important for the kind of responses I’ll be giving, I think. Now, about the graph… Everyone loves when people in the humanities delves into graphs [LAUGHTER] and math-like things. So, this should be interesting. To answer your specific question, though, much of my philosophy about higher education is rooted in work on critical pedagogy, in particular, the work of Paulo Freire and bell hooks, both of whom talk about education as a practice of freedom. And for Freire, who was deeply invested in the political climate of Brazil, education as a practice of freedom means giving students agency, empowerment, and the tools, to be able to remake the world that both benefits them, but society at large, as well. So, the liberation there comes from freeing the classroom of the kind of controlling elements that prevent that kind of learning taking place. For hooks (her writing is, as far as I’m concerned, the best education writing that has been done over the last number of decades), she takes it in a slightly different direction. And for her, the liberatory practice, education as a practice of freedom means freeing the student to be able to remake themselves and carve out a meaningful life for themselves. In order to do that, a classroom has to break down some of the hierarchies, it has to give over control to the students equally, and importantly, for bell hooks, it also means that that is a very difficult process for students, and it requires them to be vulnerable. And if we’re going to ask students to be vulnerable, the faculty member has to be vulnerable as well, or be willing to be vulnerable. And for me, in the context of grading, what that means is wrestling with the traditions that we have been handed about what education means and what teaching means. And when I refer to some of these grading practices as being more liberatory, what I mean is that by removing the emphasis on evaluation, and placing the focus on feedback, it gives over some of that control, some of that agency and empowerment, to students. So, that’s the direction that I’m moving in, and how I conceptualize it in terms of mapping the practices. Now, when I posted that on social media, it was an honest and open invitation for discussion, because where they are on that grid for me is not set in stone. In fact, I’ve had really interesting conversations about some folks who do portfolio grading… has lots of logistics. And so that was an interesting conversation. Ungrading, for me to make sense of it in my head, take some logistics to help the students feel comfortable with the process. But then I know others who have a lot of success with fewer logistics with that. But, that aside, what I’m really looking at… how much agency does the grading scheme allow students to take over for themselves?

Sarah: Lovely. Thank you.

Josh: And now, the rest of the questions are going to be kind of rooted in that project, the book about grades. And so I’m really excited to hear what you had to say about these. And the first one is kind of more of a personal question, reflecting on some of your own responses to the grades that you got when you were a student, Sarah, and how that might have an impact on the grading models you use now, as a faculty member.

Sarah: Well, I don’t think you’re gonna like my answer, [LAUGHTER] unfortunately. But I loved being graded. It was my favorite thing. So I, in middle school, and in the beginning of high school, I was always intellectually curious, but I wasn’t super into school, I would rather be buried in my books reading. And I got kind of As and Bs and occasional C in junior high. And then I started in high school, and I started racking up A’s and I was like, “Oh, I’m good at this.” So, I started getting more and more invested in my studies and more invested in grading. And I actually ended up valedictorian of my high school class. I also particularly loved… and I’ll have a question related to this later on… I loved taking exams. It’s actually the thing that I miss the most about being in college, because it was a very mastery motivated sort of thing. It was like me versus this exam. People talk about flow, and they talk about the zone, and taking an exam is where I experienced that the most. And so I had this almost gamification approach to grades. I was going to figure things out. I was going to master this thing. And I had a lot of fun with it. And then when I got to college, I was sort of in this, you know, well, I’m not going to not get an A in the class, because now I’m someone who gets As and always gets As. And that, and I’ve written about this for the Chronicle of Higher Education and writing about this in my own book on student mental health, that was part of what helped me getting over my own personal struggles with anxiety, because I wasn’t anxious about grades, and I wasn’t anxious about exams, but I was anxious for social reasons. And I was anxious about participation. And I had a really hard time participating. And I managed to get through most of my college degree without participating in class and still getting As because even though professors would say you have to participate to get an A, they would still, based on exam scores and papers, give out As. But then I started picking up women’s studies classes and these women’s studies classes had a third of the grade was participation, because it was all about discussion and all about sharing and really grappling with lived experiences. And I was kind of confronted with this, you know, am I going to not get an A? Or am I going to try to do something about my social anxiety? So, the grades were actually the motivator that helped me, not just with my intellectual journey, but also with my mental health journey. And I’m not saying that that would happen to everyone, or that my anxiety was as severe as some people’s but for me, personally, grades were fun, and they were also this motivator that had this huge effect on my life.

Josh: Great. And what you said about, particularly your approach to grades in high school, really resonated with me. I was an athlete in high school and college and really approached it in much the same way, that it was just one more competitor, one more opponent on the wrestling path…

Sarah: Yeah.

Josh: …to be conquered. But, reflecting back on it thinking, Okay, well, that obviously, there was an element of success there. But do I remember much about what I was learning in those classes? Thank you for that answer. I appreciate it.

Sarah: Well, you’re welcome. And I’m going to return to your graph, if that’s okay. [LAUGHTER]

Josh: Okay.

Sarah: So, we talked about liberation. And thank you for that beautiful answer. And I love thinking about student agency and autonomy and motivation. But, my second question is about the logistics-the other axis- and this is, in my own thinking about my own grading, something that I struggle with. And so I absolutely am in agreement, that agency and autonomy, and giving students that control, and releasing your own status are all really important. But, logistics… I think of some of these grading schemes can get kind of Byzantine and convoluted, and I have experimented with all sorts of different schemes, and I know we’re going to talk about what we’ve done in the classroom ourselves. And I think students sometimes find those really confusing, and it reduces certainty, and it reduces predictability and transparency sometimes, and they’re trying to navigate it, and they’re really struggling. And I think that that can increase anxiety more than a traditional grade structure sometimes. But then, on the other hand, the ones that are low in logistics, I have experienced student feedback where they just feel like they’re left at sea, and they don’t know where they are in the grading scheme. And most of us are teaching in institutions where, at the end of the semester, no matter what you’ve done, you have to put a grade in the system. So, certainly, it seems like liberation is good to you – more liberation is better than less liberation. How about logistics? More better, less better? Sort of your thoughts and curve?

Josh: Yeah, I think, in the way, I was envisioning that, logistics are kind of neutral. If you think about that y-axis, the liberation as being the philosophical access, the logistics is the pragmatic axis. And so, what I’m really trying to do with that particular graphic or image is to help people make decisions about what they might want to try. And, as you’re saying, for both the students and the faculty, a scheme, a model that has more logistics could be intimidating, or it could be a deciding factor. Okay, yes, I want to move in the direction of having a more progressive grading model, but doing something that is going to completely change the way I’ve approached my work for the last 20 years, and pile on top of it all these other details to think about, may just be too much. So, maybe rather than going to mastery grading, maybe I’ll start with contract grading, which can have many fewer logistics, and see how that goes. See how it aligns with my own goals and values, and pair that with how the students respond to it. In trying a number of different models, one thing is consistent and that’s what you’re saying. The students uncomfortability with changing the way they interact with grades, the way they respond to grades. So, even in the portfolio-based grading system, which to me is the baby step out of the traditional grading model, where you’re giving mostly feedback but a traditional grade at the end. I mean, the grade happens to be based somewhat on improvement over time, but still fairly traditional. Even then, there’s a lot of groundwork with students, building up their level of tolerance and comfort for that. So, yes, any shift toward a grading model that privileges feedback over evaluation is going to increase nervousness and anxiety at some level with students. And so one thing I talked about with faculty all the time is there has to be a lot of work upfront, being very transparent with students and working through this particular process. And to be honest, I don’t blame the students, because they’ve had at least 12 years of being conditioned in an educational environment where grades have meant something very specific to them. So, logistics for me, neither good nor bad, but an important factor in making decisions, both for our own workload and the student response. And I think it’s worth taking that into consideration. Because, it really matters. If we choose something right out of the gate that has a ton of logistics, and we’re trying to navigate those and help our students, it may actually discourage some people from trying out or continuing down that avenue. And that’s not what we want. And so I have known folks who have experimented, gotten frustrated, and just gone back to the grading schemes that seemed easier for them. So, that, I think is where logistics really come into this equation, both on the faculty and the student side.

Sarah: Thank you.

Josh: So, we talked about your own response to grades as a student, but what about the models that you now use as a faculty member?

Sarah: So I’ve experimented with various parts of your quadrants. And so I’ve done something in one of my classes, not purely mastery, or specifications, grading, but getting there, where there’s a lot of options, and everything is evaluated, not on a numerical score or a letter score, but instead, developing competence, achieved competence, that sort of thing. And so I’ve done kind of more high logistics versions. And then, in one of my seminar classes, which is all seniors, 15 students, and three-hour class, we’re sitting around talking about peer-reviewed neuroscience articles, I’ve taken a more purely ungrading approach. And the thing that I think I’ve decided about all of this, for at least at my institution with the courses that I teach, is that some of these techniques don’t work as well with the lower-level classes that have more content, that are more introductory, and that have younger students in them. In those classes, the students have a much more negative reaction to these untraditional schemes. They have many questions, they get frustrated. And I also, and this will lead into a question that I have for you, our students, most of them work a lot. So they’re working a lot of hours. A lot of them are commuting, they have very complicated lives. And they are not super grade conscious compared to other places that I’ve taught. They’re more focused on getting through the degree program, getting their credits, they are not “I need that A,” for the most part. And they seem to, in particular, dislike some of these grading schemes, because they want to know where they’re at. I had a student come to me, I always give a mid-semester check in and say, “Here’s where I think you’re at, where do you think you’re at?” We talk about it… but we were right before that. And she was kind of lost. And she said, “I’m working 35 hours a week. I’m taking care of my little brother. I’m taking five classes, and I just want a B in all my classes. And I have this limited pool of time. And I need to know where I should be putting my time, and I don’t know where I’m at, do I have an A minus? Do I have a C plus, I want to know if I’m in that (upper) range, because if I am I’m going to dedicate more of my time elsewhere.” And so, that was a struggle. That said, my upper-level class, I absolutely love it. So, in that seminar class, I don’t grade anything. I give them constant feedback on everything that they submit. I have tons of rubrics, and I give them lots of feedback on all different aspects. So, we’re constantly back and forth, back and forth. But we check in once in mid-semester. I ask them where they think they’re at for a grade. And it almost always matches where I think they’re at. And it just takes grading completely off the table, and we’re just talking about neuroscience, and it’s really about the intellectual discovery, and it’s not at all about evaluation. Interestingly, these students because they’re neuroscience majors, for the most part, and their seniors, they are more grade focused, but they’re more motivated and we just chat and it’s lovely.

Josh: Wow, that’s exactly what you hope for.

Sarah: Yes, [LAUGHTER] yes. I will never grade in that class again. Ever.

Josh: That’s great.

Sarah: And so I guess that leads me into this question that I have for you. So, that story of the student who just wanted to know where to divert her time and her efforts, and then my own experience with grades being sort of fun, and then also a motivation that helps me tackle some personal challenges. And then also, I think, especially in the lower-level classes, there’s a lot that is sort of just boring, rote stuff that we have to get through. And that students need that information in order to get up to those upper-level classes, and to have the foundational knowledge where they can start thinking critically, and they can start being more creative. And I think for that sort of just getting the basics down, sometimes extrinsic motivators are really valuable. And I think that the data for motivation research shows that, for creativity and critical thinking and things, we need intrinsic motivation, but for just mechanical kind of rote things, extrinsic motivation goes a long way. And so my question for you is, given all of those elements – are grades always eeevvil? [LAUGHTER] Or can they sometimes key students in to what a professor values, you know, put more effort here less effort here, motivate some of this rote learning and encourage students to face some challenges?

Josh: That’s such a good question because, I think, too often, when we begin to have this discussion in any kind of group of faculty, it’s been my experience that some people hear this as a kind of confrontation or an accusation. You give grades, grades are evil, LAUGHTER] therefore, you must be evil, which is not at all what we’re saying. What we’re trying to find here are the best approaches to help our students to learn in a meaningful way. So, I want to not flip the question, but change it just a little bit in that, for those goals that we have that are more rooted in knowledge building and rote memorization, and I agree that having some element of an extrinsic motivator can get you out of bed and into the classroom and paying attention and focus on the reason why you’re there and why you need to do well. So, I agree at that level. The question for me, especially in those lower-level courses, is what does a grade communicate? And I’m drawn to the work on inclusive pedagogy and on opportunity gaps, and what grades communicate to students who are coming to our colleges and universities from under-resourced schools where they have not had the same kind of educational opportunities. For example, to have AP classes that would prepare them, to have teachers who were invested in moving beyond just what was on the page, to have the right books, to have the right materials. And so, certainly for a subset of students from well resourced educational backgrounds, a grade in an introductory level course, could be a communicator of this information as important, pay attention to it. The grade communicates that you have mastered it well. For another subset of students from less resourced schools. I think that what the research shows is that more often than not, the grade is penalizing them for what they didn’t have, rather than being able to demonstrate what they know and can do at that point in their career. Now, there’s time over four years to bridge that opportunity gap to get them to a different place. So, what I think about this question, ultimately, in thinking through what we were just talking about with grading models, is that there’s a happy medium, where we could take an approach in introductory level classes with a mastery-based grading scheme, where knowledge standards are an important component of that grade. So, you have a whole mastery based grading, you determine the standard and the points along the way that show that students have mastered that goal. And for an intro psych course, a significant subset of those standards could be focused on knowledge and the information. And if they don’t know the difference between extrinsic and intrinsic motivation, then they haven’t checked that box. They haven’t moved their way up to mastering that particular standard. So, for me, that’s a scheme and I was just addressing this question with folks in STEM, who were talking about this very thing. People need to know information before they go out and become doctors and pharmacists and other fields. [LAUGHTER] And that’s a scheme that allows you to accomplish that goal, and to account for the educational opportunities and backgrounds of students who are coming into those introductory level classes.

Sarah: And I think part of that response makes me think that, and I know we don’t have time to unpack all of this, but the relationship between assessment and grading, and so your response makes me think more about “What kind of assessments are you using?” And of course, it’s critically tied together with grading. But, I just finished intro psych this semester, and I taught it all online. And I had previously had exams, and they were in class. And so they were by nature timed, and students had to memorize things. And because we flipped to the online environment, and I was not about to use one of those proctoring softwares, they were weekly quizzes that they had the whole day to do… very similar material. But, because it was open book and untimed, I had higher expectations. And I just calculated my grades this morning, and I had the same grade point average as last fall. And so I think that we can probably still grade, but we can make the assessing process more equitable, I think, while still using traditional grading schemes at times.

Josh: That’s a really great point. And the mastery based schemes ultimately give what look like traditional grades. But, it’s the assessments… exactly what you’re talking about…that look very different from traditional courses. You asked about extrinsic motivation. And I’m going to flip it to intrinsic motivation. [LAUGHTER] And I have some amazing colleagues in the psychology department here, who came to a recent talk that I gave on this subject. And I kind of opened with the classic research that shows that grades are an extrinsic motivator, and will impede intrinsic motivation. And for the sake of time, I was kind of glossing over some important distinctions, maybe, but they gave me some really great feedback that I’ve taken to heart and sort of built into subsequent iterations of this, which is that, “okay, but just because you get rid of grades does not mean that magically students are now intrinsically motivated,” because, as you were just talking about, some of the work in introductory level courses is really, really difficult. So, for example, they said, “You might really want to learn how to play the piano, but at the beginning, it’s really difficult.” And so, the ultimate point here, is that in order to really capitalize on the opportunities presented by decreasing the emphasis on grades for increasing intrinsic motivation, you have to have good scaffolding in there, a teacher has to come in and cultivate the intrinsic motivation, it doesn’t magically appear. So, first I wondered if you agreed with that, and second, what are some ways that we can do that work? What are some ways that we can cultivate the intrinsic motivation?

Sarah: Absolutely. And I do agree. And I think that some of the ways that cultivate intrinsic motivation is to really demonstrate how these building blocks get you to really interesting places. LAUGHTER] And one example I had was in one of my first speaking engagements. And someone there shared the story that they were in a nursing program, and some of their introductory courses were very difficult and very rote. And the students were a long way from being nurses, and that they were having trouble motivating their students. And what they did, which I thought was wonderful, is they had their students who were in internship or placements. And in their senior year, or right after their senior year, they had them come back to their class and do a whole class on specifically what they were doing now, which was very interesting, working with patients and all the things that the students were looking forward to. But then they shared explicitly how they were using the information that they were learning in these introductory classes. And so this very real life demonstration of “Yes, this is kind of a slog right now, and I know it’s a struggle. But here are the ways that mastering this content is going to get you to this place.” And I think we talk a lot about various forms of representation. But, like having someone that you can visualize, “Oh, this is me in four years, and here’s how they are using this material in pursuing the goal that I want to pursue…” I thought it was a really beautiful example of doing some of that scaffolding. And even if you don’t have access to that sort of setup, I think that ways of not just staying in the simple foundational knowledge, but showing how “Oh, this is how this informs this current controversy, or this informs these decisions that are being made in the real world,” I think, are ways to demonstrate the intrinsically interesting aspects of this foundational knowledge.

Josh: Great.

Sarah: So, as you know, my new project is all focused on the best learning environments for student mental health, in particular. And you have one very successful book on how human beings learn and are working on these new projects on anxiety and grading. You direct your Center for Teaching Excellence and you get talks all over the country. So you are an ideal person to ask: “What do you see as the best learning environment to both help students learn and enhance their well being and mental health?”

Josh: I can’t wait to read this new book that you are working on. I think it’s gonna be great. So, what I keep coming back to the area of research that you and I both really love and keep coming back to, and that is emotion, and the environments that I see my own classrooms, my own university, but many places elsewhere, always come back to care and compassion. That over and over again, the learning environments that are most successful for students are those where they feel as if the instructor cares about their learner. And that can look wildly different depending on who the faculty member is. But the student response tends to be the same. I felt invested, because it was clear that the instructor really cared that I succeeded in this class. And you were just saying… Sarah gave a great talk at the University of Mississippi on Tuesday. And she was saying in that talk, and I thought it was a great point, that whenever you move into this discussion, an automatic reaction is “This is getting into touchy feely territory. Right?” [LAUGHTER] That’s a dominant response. And I understand that response. But that is not what we’re talking about. We’re simply talking about having an environment where students feel respected, where they feel valued, where they feel as if their success matters. And sadly, I think in a lot of learning environments, they don’t feel that. So it’s surprising to them when they are in a class that does allow them to feel those emotions. On the faculty side of it, and this is where I think this automatic response comes from, it doesn’t mean you have to develop friendships, although sometimes down the road that happens, and I still keep in touch with students I had 10 years ago, and they’re doing amazing things. But, that’s not what we’re talking about. You can keep all the professional boundaries in the world that you want, and still show students that you are there to help them succeed. Some of the research that I admire the most in the area of social belonging, Geoffrey Cohen and Gregory Walton have a wonderful paper that showed the amazing benefit of a simple comment at the top of a student’s paper. So one group of students only got targeted feedback, another group got targeted feedback plus a comment at the top that said, “I’m giving you this feedback, because I have high standards, and I have every reason to believe that you can meet those standards.” And that had ripple effects that they traced over years in the success of those students academically over time. And while that may seem extraordinary, the sentiment is not. In practice, that one simple way of communicating to a student that the faculty member is here to help you succeed. I’m not here to hear myself talk, I get tired of that, right? [LAUGHTER] I’m not here for me, I’m here for you. And that is the common denominator that, at least, I have observed, both in practice and in research as being the most beneficial thing for helping students in that way.

I have a question that I just thought of this morning. So, I hope you’re okay with some improvisation. It’s a general question, though. So no preparation needed. It’s in two parts. So, maybe if we take the first part, and then come back to the second one? The type of writing that you’re doing now, Sarah, is not… I wouldn’t say very different, but it isn’t necessarily traditional academic writing. It’s popular scientific writing. It’s for a general educated audience. And so I was wondering if you could just talk about how you made that transition as a writer and some of the strategies that you use, I think the audience may really value hearing about that process for you and the evolution of your time.

Sarah: Well, I had done quite a lot of academic writing. And then my first book… Jim Lang approached me about writing a book for his series, because I had been blogging. So, I guess that predated it. So I’d been doing some blogging, first for a Martha Stewart publication, and then for Psychology Today, and he was looking for people who could write accessibly about cognitive science. And he said to me that most social scientists write like robots. [LAUGHTER] And he said, “I don’t want my books to read like they’re written by robots. And so would you be willing to do this?” And so I think that part is just stopping writing like a robot and picturing your audience. You know, a lot of writing advice talks about picturing your audience and writing for an audience, and I think that that’s true. But I think what really is the answer is that I write in my teaching voice. And the voice that I have, I sometimes don’t like my writing voice, I think gets a little chatty, but I tap into the same part of myself when I teach. And so I think that it’s: how does this material relate to my own personal experiences? to things going on in the world? to real-life phenomena? What are metaphors that I can use? What are anecdotes that I can use to illustrate this? What are ways that I could do this in a way that’s kind of positive and hopeful? And all of these things are things that I try to do when I teach. And Hivemind, my second book, it had a purpose, but a lot of it was: here are the coolest things that I’ve been sharing in my classes for the last 10 years, my favorite neuroscience studies, my favorite psychology studies. And then there’s certainly an overarching framework. So, that was even more clearly my teaching voice in writing form.

Josh: I love one of the ways that Jim talks about writing is “the voice of the colleague down the hall.”

Sarah: Mm hmm.

Josh: And I think that you do that so well. He does that so well. And it’s really accessible. And I think an important development in books about teaching.

Sarah: Well, thank you.

Josh: The second part of this question, though, is a little bit more technical, but I think people might benefit from hearing about it. And that is, the sort of interviews that you and I are doing right now, that you did a lot in Hivemind, that you’re doing in your book, are different from the kinds of interviews that social scientists would do in peer-reviewed publications, because we’re not trying to use the interviews to make a research claim, we’re using them more in an illustrative sort of way. Here’s our point. Here’s someone who’s doing that thing, or thinking about that thing, and illustrating that. And so my kind of technical point that I think is worth thinking about, what is the role of your IRB in doing this kind of writing? Because I know, on my campus, they want me to submit about this project, and talk about the fact that the interviews look more like what a journalist would do than what a social scientist would do. So, that they can say this does not actually qualify as research. It’s just kind of a box checking, so that they have it on file, and they know what’s going on, and that makes a lot of sense to me. But, I’m wondering, kind of from a technical standpoint, what do you have to do at your campus to make this writing work for your school and your career?

Sarah: Well, until recently, I was chair of my IRB. [LAUGHTER] So, I don’t know if that taints my response, but it’s two different things. So, the interviews like this, I’m actually a little surprised to hear that your IRB had you submit. It sounds like an exempt, just checking that it’s exempt from IRB review, which is not IRB review. But, actually, the Common Rule states that journalism and oral history projects and a couple other sorts of categories of doing interviews, they’re not considered research. Research has this technical definition of data gathered in order to contribute to generalizable knowledge. And because things like oral histories and journalism are on specific topics and they’re opinions about things, they are not considered research. And so, on my campus, they don’t go through IRB review. The exception is, so in my new project, unlike my older projects, I am also doing student interviews, and that is actually not even exempt from IRB review. It’s gone under IRB review. And so that part of the project, I did submit for official review, and a number of committee members reviewed it and approved it. And so the student data, and that will be anonymous… The student interviews were considered research. were reviewed by the IRB, even though it’s for kind of a journalistic book project, but the expert interviews that I’m doing, I did not.

Josh: I just think it’s interesting, too, as we think about doing different types of writing, how our campuses see that, and what kind of role they play in that, as well. So thank you.

Sarah: So what’s next, Josh?

Josh: A lot of writing is next, I think. [LAUGHTER] And you know how this is, these are writing projects that require a lot of mapping out ahead of time and squeezing that into campus responsibilities, like workshops, and things like that. But, I’m looking forward to the process. It’s always fun to be kind of starting out on something new. Sarah, what’s next with you?

Sarah: Pretty much the same. [LAUGHTER] I’m going to be doing a lot of interviews. So, people listening to this may be getting little taps on their shoulders. And I am also launching the student interview portion, which I’m excited about, because I’m eager to hear what they think. This is my first qualitative study I’ve ever done. So, it’s going to feed into the book project, but it’s also hopefully going to be a peer-reviewed article. And I’m working on that with my honors student, Jasmin Veerapen, and she and I are presenting at POD together. So, people who go to POD, you can check that out. And I’m eager to see what students have to say about all of this, because we’re going to try to let the data speak to us, let the students speak to us. And so we have hypotheses in our head, but we want to see where they take the conversations.

Josh: That was an amazing conversation.

Sarah: This has been fun. Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Brittany Jones and Savannah Norton.

[MUSIC]

155. Remote Proctoring

Faculty who rely on high-stakes proctored exams in their classrooms often attempt to replicate this approach in online instruction by using remote proctoring services. In this episode, Jessamyn Neuhaus and John Locke join us to discuss some of the issues associated with the use of remote video proctoring and suggest some effective and less problematic alternative methods of assessing student learning.

Jessamyn is the Interim Director of the SUNY Plattsburgh Center for Teaching Excellence and a Professor in the History Department at Plattsburgh. She specializes in the study of pop culture, gender studies, and teaching and learning. Jessmyn is the recipient of the State University of New York’s Chancellor’s Award for Teaching Excellence. She is also the author of Geeky Pedagogy: a Guide for Intellectuals, Introverts and Nerds who Want to be Effective Teachers. John is the Coordinator of Technology Enhanced Learning and an adjunct instructor in Communication Studies, also at SUNY Plattsburgh. He recently received his doctorate in interdisciplinary studies with a concentration in humanities and culture, and is currently working on a second historical novel.

Show Notes

Additional Resources/References

Transcript

John: Faculty who rely on high-stakes proctored exams in their classrooms often attempt to replicate this approach in online instruction by using remote proctoring services. In this episode, we discuss some of the issues associated with the use of remote video proctoring and suggest some effective and less problematic alternative methods of assessing student learning.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Our guests today are Jessamyn Neuhaus and John Locke. Jessamyn is the Interim Director of the SUNY Plattsburgh Center for Teaching Excellence and a Professor in the History Department at Plattsburgh. She specializes in the study of pop culture, gender studies, and teaching and learning. Jessmyn is the recipient of the State University of New York’s Chancellor’s Award for Teaching Excellence. She is also the author of Geeky Pedagogy: a Guide for Intellectuals, Introverts and Nerds who Want to be Effective Teachers. John is the Coordinator of Technology Enhanced Learning and an adjunct instructor in Communication Studies, also at SUNY Plattsburgh. He recently received his doctorate in interdisciplinary studies with a concentration in humanities and culture, and is currently working on a second historical novel. Welcome, John, and welcome back, Jessamyn.

Jessamyn: Thank you. Thanks for having us.

John L.: Yeah, thanks.

John: Today’s teas are:

Jessamyn: Just plain water for me. Gotta stay hydrated.

John L.: Grande decaf from Starbucks.

John K.: That’s an interesting tea.

Rebecca: Yeah. [LAUGHTER] I have a Scottish afternoon tea

John K.: …and I have ginger peach green tea.

We’ve invited you both here to talk about online proctoring services. As a result of the global pandemic, a lot of people suddenly had to shift from face-to-face instruction to remote instruction or online instruction. And many people who relied on proctored classroom exams are concerned about how to offer tests, and many faculty have been investigating the possibility of using remote proctoring services. What are some of the concerns associated with using online proctoring services?

John L.: Well, to start with, we are all trying to deal with the digital divide. And when you get into online proctored exams, that becomes a pretty big issue in that not all students have the equipment or the bandwidth to be able to participate. It helps to know what the process is. And basically, what we’re dealing with is a test that’s happening while the student is being recorded, both audio and visually being recorded. Usually, it starts out with a little intro section where you have to show an ID to prove who you are, show your space so that everybody can see that you don’t have crib notes on your desk, or there isn’t Albert Einstein in the corner of the room [LAUGHTER] telling you the answers to what you’re working on. And assuming all that goes well, then, of course, you’re taking the tests, usually an online test with a lockdown browser so that you can’t surf for answers anywhere else. It’s a lot of moving parts to make it work in the first place. And the big assumption is, number one, the student has the equipment necessary, and the student has the environment necessary to take a quiz like that. For instance, if you happen to be a student who lives in a very small apartment with a family, and you have brothers and sisters running through the room where you’re taking the test, because you’re at the dining room table, there are so many issues that come into play, not to mention just the fact that you may be embarrassed by your surroundings and don’t feel comfortable showing those surroundings to other people. So for me, that’s probably the first and most critical reason why I always talk to faculty and ask them to think about it before they actually devote themselves to that process. Other issues are, try as you may, there are always ways to get around these sorts of safeguards. And if there’s one thing we know, it’s that somebody who plans to be dishonest will figure out a way to be dishonest. Again, I try to get instructors to be a little more thoughtful with how they’re going to assess that learning is taken place in the first place. And that’s really where my friend Jessamyn has opened my eyes to many of the alternative ways.

Jessamyn: Yeah, there’s a lot of great resources that have been proliferating since the emergency pivot in response to this very question and suggestions, building on research that was already there, for how to assess student learning and in authentic and, as John was mentioning, equitable as possible way. I guess, just what I would add to that in terms of looking at it as a scholar of pedagogy, and taking messages like from James Lang’s book, Cheating Lessons, what do you want to foreground in your message to students in the class climate you’re creating, in the rapport that you’re building with them? The ordeal of the kind of proctoring software that John was describing, and that we were increasingly seeing problems with… the very first message you’re sending to students is: I assume students cheat, I assume students are going to be dishonest. I assume students don’t care about their education enough to try to express their learning as honestly and authentically as possible. And I guess what we, as what John and I both, were inviting faculty to consider when we were doing workshops this summer on this topic is: are there alternatives to this that send a more positive message and create a more productive class client and help you connect to students? Let’s not forget, at a time when everybody is anxious and overextended and fearful, we’re still in the middle of a pandemic. So, what do you want to prioritize as an educator?

John L.: Yeah, and exams are stressful enough as it is. So you add COVID on top of that, and then you add a technology that students aren’t used to. And it’s so much easier to choke under that environment.

Jessamyn: Yeah, an anxious brain is not a brain that can clearly and, to its best ability, express what it knows and show what it knows. All the information about trauma-informed teaching just reminds us that if every chemical and message in your brain is saying, “Run away from the tiger that’s hiding in the jungle,” there’s no room to: “Okay, move your webcam to show behind your ears that you don’t have an earpiece. Now take your laptop over to the door and show that it’s closed.” How is that not creating a prey state of mind with the predator waiting to pounce on you?

John K.: Each of the issues that you both talked about also have a very differential effect in terms of creating an inclusive classroom environment. People from high-income households are more likely to have some nice quiet space, are likely to be able to afford equipment that will work with proctoring software, while Chromebooks and most mobile devices will not work well with proctoring services. And also issues of anxiety and concern about being successful are also probably more likely to be experienced by students who are first- gen students who don’t necessarily have the same expectations of being successful based on their family environment and their social networks. One of the things that concerns me about all this is that the impact would be differentially imposed on students who are already at a disadvantage in terms of the quality of their prior schooling and their resources and their support networks.

John L.: Yeah, I couldn’t agree more. I’m not sure what to add to that, John. [LAUGHTER]

Jessamyn: I can jump in though. I had a thought. I’ve been reflecting… I can’t get it out of my head from a webinar this week that the Chronicle of Higher Education did a panel about the human element in online learning. And one of the panelists, Viji Sathy, mentioned that this crisis has really brought home to a huge new number of educators that we are teaching whole students… that taking into account all aspects of students experiences, their work experiences, family experiences, and these equity issues. So, it’s not that academic inequality is brand new to 2020. But, the awareness of it has really increased and the attention to it has really increased. And I think it’s being highlighted in ways that it’s just impossible to look away from. So this specific issue is touching on, I think, a bigger kind of reckoning that faculty are having on an individual basis, and as institutions. I see a lot more individual instructors really asking, “Wait, am I being inclusive?” The question is way more in people’s minds than I think it’s ever been, in my experience.

Rebecca: Related to that is the idea of accessibility too. With so much delivery in digital formats, the topic of digital accessibility is becoming much more prevalent in the forefront of faculty’s minds, whether they want it to be or not, it becomes something that everyone’s becoming more aware of. This same kind of software also imposes a lot of accessibility issues and barriers for students with disabilities, because a lot of them are not compatible with assistive technology and aren’t built to Web Content Accessibility Guidelines, essentially.

Jessamyn: And related to that, students with anxiety issues, who are struggling with mental health issues… the high-stress, high-stakes examination, in any format, is a challenge. But add to that the technology aspect of it, you are looking at assessment mechanisms that really isn’t being accessible and inclusive, it would not allow all your students to show you what they know.

John K.: One concern that I have about proctoring services is that faculty may see it as a simple solution that will allow them to use tests that they’ve created in the past. Many people have created very elaborate test banks in Blackboard and other places and then they expect that those questions can now be used, if they’re used in a proctored environment, not realizing that most of those questions have already been distributed to multiple sites out there and students would often have access to them, anyway. So I think that proctored systems can provide instructors with a false sense of security and as John mentioned earlier, they can be pretty easily defeated as long as students have devices that will allow them, for example, to do screen shares in the background underneath the proctoring service or perhaps have multiple devices where they can be looking up answers or using some other mechanism that won’t always be easily detected by the proctoring service.

Jessamyn: That’s a good point, and I know John Locke has addressed that issue. I mean, you don’t drill in on it, but when you’re talking to faculty, you often say, “And by the way, this is not a magic bullet, even if you go through all the trouble of setting it up.”

John L.: The idea that somehow having someone else proctor your exam is going to save you time…. That’s not how it works. These proctoring systems just flag potential incidents. You still have to go through and you decide whether or not those are warranted as cheating or if they’re just someone sneezed. So, between setting up the exam and then reviewing the flags, looking for false flags, I don’t know if it saves anybody any time.

Rebecca: I’m team workload reduction.

Jessamyn: Yes.

John L.: Yeah. [LAUGHTER]

Rebecca: So, what do we say to faculty who ask about replicating those high-stakes testing environments in their online environments.

John L.: I say: “Why?” I think that what would be more appropriate is to simulate the environment that somebody needs to perform in where they’ve acquired the knowledge in order to accomplish that performance. For instance, I taught a computer applications course years ago, and for the final exam…. I did have a final exam… but, I told them, “What I hope you get out of this class is to learn how to learn how to use software. So if you haven’t already learned how to learn to use software, now’s your chance. And when you’re out in the real world, you will have the software manual, you will have the person in the cubicle next to you, the only thing you won’t have is me. So, unless you have a question about a specific question on the test, don’t talk to me, as far as anything else that gets you to accomplish the goal, go for it.” If you’re studying to be an ER doctor, perhaps you do need to have the pharmaceutical manuals memorized page by page. [LAUGHTER] But most of us aren’t working in that kind of stressful environment. So, there are better ways, maybe project-oriented ways, to assess that that learning has taken place, that those skills have been received or learned and received.

Jessamyn: I try to assume best intentions on the part of all faculty. And I know that many of my colleagues who expressed that sentiment exactly, like “How can I make sure they’re not cheating?” …they’re not saying that because they’re evil, like “Mwah, hah hah hah, those bad students…” No, they really are concerned about student learning. So, what John and I did was really to frame this as an invitation to faculty, an invitation to think creatively about assessment, authentic assessment, to really be able to measure student learning, but maybe also rethink what you thought and assumed about assessment. And here’s a big bonus, maybe grading it could be less painful. If you are trying something new, something that’s a little bit more creative, that might help you as well in your end. So, that’s been how we’ve been addressing it here at Plattsburgh.

Rebecca: What are some ways to do that assessment, maybe in a class that doesn’t work well for project-based learning. Maybe it’s a bigger section class, or maybe it’s more foundational information that doesn’t lend itself as easily to project-based learning. What are some alternatives?

Jessamyn: There’s always small, lower stakes, regular quizzes. So instead of one big, huge exam, having smaller quizzes along the way. That’s just one off the top of my head… an easy one. John?

John L.: Yeah, well, especially in this environment, discussion forums are really, I think, underutilized. There’s no reason that you can’t build a rubric around a discussion forum and spell out your expectations to students and then hold them to them and grade according to those. Again, it’s taking the student higher up that Bloom’s taxonomy ladder than just memorizing and regurgitating information. It’s causing them to react to other people’s comments within the discussion forum, to assimilate the knowledge that they’ve already accumulated, and to create new and different responses based on that immediate situation. And, the advantage to that for slow thinkers like me, [LAUGHTER] is that you don’t have to be quick on your feet. You’re not the student in the back of the room with his hand up saying, “Well, never mind, you covered that five minutes ago.” It’s kind of an equalizer. I wouldn’t say “Have a discussion forum as a final exam,” but it’s another part of the scaffold to assess that learning is taking place throughout the semester.

Jessamyn: I think there’s a lot of potential for open-book exams as well. In fact, I have used open-book exams for a long time. And, in large part, that is because I really wanted my students to learn, and I wanted to be able to grade an exam very rigorously. So saying, here’s a question you can answer with an open book, and, yeah, you might even talk to someone else about it. But then the final product is an essay question, or it could be a presentation, it could be a sort of annotated bibliography. There’s lots of ways it could go as an open-book exam. But then when I go to assess it, I know that you have the material in front of you. So, I am going to really drill down here, like, “Do you really understand this concept? Can you show me that you understand it?” Because I know you can look at the basic definition in the book that’s open in front of you. So, now you have to show me that you really, really get it, you have to use it, you have to apply it, whatever it is.

Rebecca: What about STEM-oriented examples? A lot of the things that we’ve talked about work really well in the humanities and the arts. How about some things that work well in math and science and other STEM fields?

Jessamyn: So, I’ve been trying to do a little reading in this area. I’ve been hearing from some faculty in this area. So, in an online lab setting, being able to complete the experiment in the correct way, in the scientific-y way… [LAUGHTER] …that could be one way to assess learning… doing something like a fact sheet. So the final product is how you’re assessing the student learning. But again, you could be measuring the application, the correct way to do XYZ in a kind of fact sheet format or a PowerPoint slide or a poster presentation.

John K.: One type of thing we sometimes recommend for people in the STEM fields is that, if they are going to use multiple choice, one way of dealing with this is to use some algorithmically generated questions so that each student gets their own version of the question. Now, the solution procedure may be the same, but for at least low-level skills, that can help to deter some academic integrity issues.

Jessamyn: Student-generated exam questions could be another way to go. If you really understand the material, you’re not just regurgitating memorized material, but if you really understand it, then you should be able to help someone else understand it. And one way you could assess that would be “What are the 10 best exam questions?” …something like that.

Rebecca: Another idea that I’ve heard from people more in the STEM areas is the idea of creating some sort of resource that explains a topic to a non expert audience. So, maybe it’s an experiment or something that you can do with kids, or just kind of generally to someone who’s not in the discipline and get them to grasp whatever it is that you’re trying to assess.

Jessamyn: Yeah.

John L.: This might be going out on a limb for a STEM environment, maybe we could call it STEAM, because there is an artistic bent to it. But, for instance, in an accounting course, if there’s a particular accounting procedure or process that students have to prove that they understand it, they could write a short story, “a day in the life of the accountant to the New York Yankees” or something… and totally fictional, but covering each step in the process that has to be accomplished. And as an instructor, I would love to read something like that rather than checking off right or wrong on a test sheet.

Jessamyn: I’m thinking too about something like following up Rebecca’s suggestion, and increasing accessibility, you could even have students creating resources like that in a variety of formats. It could be a poster, could be a podcast, could be a video, could be a live presentation… You could do something like an oral exam… something like that.

John K.: One of the things I’m doing in my small class of 60 students is having students create podcasts. Unfortunately, that doesn’t scale as well, in my class that’s closer to 300 students. So, I’d really like to do more open pedagogy projects. It’s just, in large intro classes, that’s a bit of a challenge.

Rebecca: John, you have some experience using algorithmic questions, too, as a way of assessment, right?

John K.: Algorithmic questions can work very effectively, in at least making sure that students can use the formulas appropriately, which is a basic skill in many STEM classes.

Jessamyn: What I would like to see is more faculty really having these discussions and swapping these ideas, like on a national scale. I think that the learning curve has been so high for so many instructors in so many ways. Like, not just, “I’ve never even visited the learning management system, and now I have to use it.” Not just that. But, coming to terms with the emotional aspects of teaching and trauma-informed teaching in the midst of, possibly, “I’m at home and I’m supposed to be overseeing my children’s education” or simple childcare issues. All these things are overwhelming so many instructors just day-to-day life. And then on top of that, “Oh, rethink something you’ve used forever. The thing that you relied on from day one, and that you did so well in graduate school… hey, that’s not gonna work.” That’s hard. That’s tough. So, the more sharing of ideas we have, and the more spreading of good possibilities for assessment, the better. And I sent you a list of some of those resources I’ve been providing. They are starting to be generated, especially at university teaching centers and in people’s blogs and essays and such. But, I think the more it just becomes a broad conversation about “What can we do? How can we, in this situation, assess student learning in new ways and recognizing it’s new for us, too.”

Rebecca: Bill Goffe, in our episode 154, Sharing Disciplinary Pedagogies, also offered a way to get people to collaborate across institutions on some of these kinds of things using a simple Google Sheet. So, we’re all kind of forced to be on line in some capacities now, maybe more than before, but maybe that’s also opening some doors for collaboration that haven’t been there before, either.

Jessamyn: I hope so. I mean, John Locke and I, both of our centers had not been collaborating in the past. So, spring of 2020, was like this kind of completely perfect context for us to send a message to the university, the Center for Teaching Excellence, and Technology Enhanced Learning, we work together, and because people needed us both. So, in that sense, I won’t say silver lining, there’s no such thing right now, but it was a unique opportunity for these two very small centers on campus to collaborate.

John L.: Yeah, in fact, I’ve accidentally come up with a tagline that is starting to appear at the bottom of my emails to faculty. And that is, “you are not alone.” They never were, but it’s much more important for them to realize. In fact, I was working with a professor last night who was having some difficulty in the learning management system. And about 10 o’clock, I sent him what I thought was probably the solution. And I didn’t hear back. So, this morning, I sent him an email and said, you know, “How did it work out?” And his response was, “I’m sorry, I haven’t even gotten to it yet. I’m sorry.” And I said, “No, you don’t have to apologize to me, I just want you to know that you’re not alone, that I’m trying to help you. And I’m not going to let go until I know your problem is solved.” And that sort of community approach to learning in general, and what we’re all going through, I think is helpful. If you know that I know I’m struggling with this I’ll bet someone else is too and, maybe between us, we can figure it out. If more people can adopt that thought and not feel that they’re infringing on someone else’s time, I think we’ll all get through this to whatever the other end looks like.

Jessamyn: That was one of the first things that John Locke has said to faculty who wanted to use this remote proctoring system is “Don’t make your life harder than it has to be.” All the student issues aside, and equity and trust and accessibility, but it’s such a pain in the ass. It really is hard to use. And I’m not just talking to the student end is terrible, but from the instructor end. It’s such a pain to set up and he shared with me, sometimes someone will approach him, “Can I set this up,” he said “Okay, but you have to do bla bla bla bla bla, then this and this…” and they’re like, “uh, maybe I’ll rethink this.” LAUGHTER] I mean, let’s try to make our teaching a little bit more joyful, if we can. Let’s try to make it a little bit more creative, for our sake, if nothing else,

John K.: It can be a lot more fun listening to podcasts students create, listening to their videos that they create, looking at documents they create, or infographics and other things, than it is reading a pile of exams, or writing up multiple choice exams.

Jessamyn: For students, too. Conveying their knowledge in a different way. It’s so good for their brain. That’s why I’m always reassuring students, when I’m asking them to do non- traditional assessments, which I mostly use (even before all this). Our students are very traditional in many ways, and they get really nervous when I say, “Okay, so you’re gonna write a short story, you’re gonna do a poster.” And they say: “Wait, what? I’ve never done that before.” Or “ I don’t know, I don’t know if I can do that successfully.” And I’m constantly telling them, “This is you conveying your learning, your skills, your knowledge in a new way, and it feels challenging, but you could do it and it’s great for your brain. It’s like calisthenics for your brain. You’re presenting what you know, just like you would in a traditional research paper or a traditional exam, but it’s in a different format, and that’s great for your thinking in all ways.”

John K.: We always end with the question, what’s next?

John L.: What’s next? I’m waiting for that chip to be implanted in my head so that I won’t have to show you my assessment, you’ll just be able to download it. [LAUGHTER]

Jessamyn: John, what is your next book project?

John L.: My next book project… I’m writing a novel that’s called “Defending Eldorado” and it takes place in South America, about 50 years after Columbus, where a bunch of colonial powers are trying to find Eldorado and the native South, Central and North Americans are doing their best to make sure they don’t find it. And since we never did, obviously, they were successful. Spoiler alert. [LAUGHTER]

John K.: You mentioned that you had just completed a book. What was your most recent book about?

John L.: Ah, my most recent book was actually the prequel to the current book, a nd that was about a group of disillusioned European scholars who left the Academy. They were humanists, they left the academy because it was being run by scholastics. And they decided to find Thomas More’s Utopia, which leads them to the New World, and hilarity ensues. Not really, but… [LAUGHTER]

Rebecca: How about you, Jessamyn?

Jessamyn: I’m headed, coming up, very shortly, I think everybody here is familiar with it, the SUNY Faculty Developers Conference, it’s going to be online and I’m doing a poster there about a series of events that John Locke and I hosted over the spring for faculty. So, that’s coming up next month. I’ve got some speaking things coming up. I’m really excited to be speaking at the Lilly Online Conference in November, and I am reading chapter submissions for an anthology project that’s contracted with West Virginia University Press in their Teaching and Learning Series. It is an anthology of insights into effective teaching and learning from women, marginalized, and underrepresented faculty. I have some fantastic submissions… so many good ones. So, that’s been a really great thing I’ve been working on right now. It’s fun.

Rebecca: Well, thank you both for joining us…

Jessamyn: Thank you.

Rebecca: …and we look forward to your future work, for sure.

John L.: All right, thank you.

John: It’s great talking to both of you.

Jessamyn: Nice to see you both. Hang in there, SUNY Oswego.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Brittany Jones and Savannah Norton.

[MUSIC]