155. Remote Proctoring

Posted in access, accessibility, assessment, authentic learning, edtech, equity, higher education, inclusion, learning, online learning, pedagogy, project based learning, social justice, teaching
September 30, 2020

Faculty who rely on high-stakes proctored exams in their classrooms often attempt to replicate this approach in online instruction by using remote proctoring services. In this episode, Jessamyn Neuhaus and John Locke join us to discuss some of the issues associated with the use of remote video proctoring and suggest some effective and less problematic alternative methods of assessing student learning.

Jessamyn is the Interim Director of the SUNY Plattsburgh Center for Teaching Excellence and a Professor in the History Department at Plattsburgh. She specializes in the study of pop culture, gender studies, and teaching and learning. Jessmyn is the recipient of the State University of New York’s Chancellor’s Award for Teaching Excellence. She is also the author of Geeky Pedagogy: a Guide for Intellectuals, Introverts and Nerds who Want to be Effective Teachers. John is the Coordinator of Technology Enhanced Learning and an adjunct instructor in Communication Studies, also at SUNY Plattsburgh. He recently received his doctorate in interdisciplinary studies with a concentration in humanities and culture, and is currently working on a second historical novel.

Show Notes

Additional Resources/References

Transcript

John: Faculty who rely on high-stakes proctored exams in their classrooms often attempt to replicate this approach in online instruction by using remote proctoring services. In this episode, we discuss some of the issues associated with the use of remote video proctoring and suggest some effective and less problematic alternative methods of assessing student learning.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Our guests today are Jessamyn Neuhaus and John Locke. Jessamyn is the Interim Director of the SUNY Plattsburgh Center for Teaching Excellence and a Professor in the History Department at Plattsburgh. She specializes in the study of pop culture, gender studies, and teaching and learning. Jessmyn is the recipient of the State University of New York’s Chancellor’s Award for Teaching Excellence. She is also the author of Geeky Pedagogy: a Guide for Intellectuals, Introverts and Nerds who Want to be Effective Teachers. John is the Coordinator of Technology Enhanced Learning and an adjunct instructor in Communication Studies, also at SUNY Plattsburgh. He recently received his doctorate in interdisciplinary studies with a concentration in humanities and culture, and is currently working on a second historical novel. Welcome, John, and welcome back, Jessamyn.

Jessamyn: Thank you. Thanks for having us.

John L.: Yeah, thanks.

John: Today’s teas are:

Jessamyn: Just plain water for me. Gotta stay hydrated.

John L.: Grande decaf from Starbucks.

John K.: That’s an interesting tea.

Rebecca: Yeah. [LAUGHTER] I have a Scottish afternoon tea

John K.: …and I have ginger peach green tea.

We’ve invited you both here to talk about online proctoring services. As a result of the global pandemic, a lot of people suddenly had to shift from face-to-face instruction to remote instruction or online instruction. And many people who relied on proctored classroom exams are concerned about how to offer tests, and many faculty have been investigating the possibility of using remote proctoring services. What are some of the concerns associated with using online proctoring services?

John L.: Well, to start with, we are all trying to deal with the digital divide. And when you get into online proctored exams, that becomes a pretty big issue in that not all students have the equipment or the bandwidth to be able to participate. It helps to know what the process is. And basically, what we’re dealing with is a test that’s happening while the student is being recorded, both audio and visually being recorded. Usually, it starts out with a little intro section where you have to show an ID to prove who you are, show your space so that everybody can see that you don’t have crib notes on your desk, or there isn’t Albert Einstein in the corner of the room [LAUGHTER] telling you the answers to what you’re working on. And assuming all that goes well, then, of course, you’re taking the tests, usually an online test with a lockdown browser so that you can’t surf for answers anywhere else. It’s a lot of moving parts to make it work in the first place. And the big assumption is, number one, the student has the equipment necessary, and the student has the environment necessary to take a quiz like that. For instance, if you happen to be a student who lives in a very small apartment with a family, and you have brothers and sisters running through the room where you’re taking the test, because you’re at the dining room table, there are so many issues that come into play, not to mention just the fact that you may be embarrassed by your surroundings and don’t feel comfortable showing those surroundings to other people. So for me, that’s probably the first and most critical reason why I always talk to faculty and ask them to think about it before they actually devote themselves to that process. Other issues are, try as you may, there are always ways to get around these sorts of safeguards. And if there’s one thing we know, it’s that somebody who plans to be dishonest will figure out a way to be dishonest. Again, I try to get instructors to be a little more thoughtful with how they’re going to assess that learning is taken place in the first place. And that’s really where my friend Jessamyn has opened my eyes to many of the alternative ways.

Jessamyn: Yeah, there’s a lot of great resources that have been proliferating since the emergency pivot in response to this very question and suggestions, building on research that was already there, for how to assess student learning and in authentic and, as John was mentioning, equitable as possible way. I guess, just what I would add to that in terms of looking at it as a scholar of pedagogy, and taking messages like from James Lang’s book, Cheating Lessons, what do you want to foreground in your message to students in the class climate you’re creating, in the rapport that you’re building with them? The ordeal of the kind of proctoring software that John was describing, and that we were increasingly seeing problems with… the very first message you’re sending to students is: I assume students cheat, I assume students are going to be dishonest. I assume students don’t care about their education enough to try to express their learning as honestly and authentically as possible. And I guess what we, as what John and I both, were inviting faculty to consider when we were doing workshops this summer on this topic is: are there alternatives to this that send a more positive message and create a more productive class client and help you connect to students? Let’s not forget, at a time when everybody is anxious and overextended and fearful, we’re still in the middle of a pandemic. So, what do you want to prioritize as an educator?

John L.: Yeah, and exams are stressful enough as it is. So you add COVID on top of that, and then you add a technology that students aren’t used to. And it’s so much easier to choke under that environment.

Jessamyn: Yeah, an anxious brain is not a brain that can clearly and, to its best ability, express what it knows and show what it knows. All the information about trauma-informed teaching just reminds us that if every chemical and message in your brain is saying, “Run away from the tiger that’s hiding in the jungle,” there’s no room to: “Okay, move your webcam to show behind your ears that you don’t have an earpiece. Now take your laptop over to the door and show that it’s closed.” How is that not creating a prey state of mind with the predator waiting to pounce on you?

John K.: Each of the issues that you both talked about also have a very differential effect in terms of creating an inclusive classroom environment. People from high-income households are more likely to have some nice quiet space, are likely to be able to afford equipment that will work with proctoring software, while Chromebooks and most mobile devices will not work well with proctoring services. And also issues of anxiety and concern about being successful are also probably more likely to be experienced by students who are first- gen students who don’t necessarily have the same expectations of being successful based on their family environment and their social networks. One of the things that concerns me about all this is that the impact would be differentially imposed on students who are already at a disadvantage in terms of the quality of their prior schooling and their resources and their support networks.

John L.: Yeah, I couldn’t agree more. I’m not sure what to add to that, John. [LAUGHTER]

Jessamyn: I can jump in though. I had a thought. I’ve been reflecting… I can’t get it out of my head from a webinar this week that the Chronicle of Higher Education did a panel about the human element in online learning. And one of the panelists, Viji Sathy, mentioned that this crisis has really brought home to a huge new number of educators that we are teaching whole students… that taking into account all aspects of students experiences, their work experiences, family experiences, and these equity issues. So, it’s not that academic inequality is brand new to 2020. But, the awareness of it has really increased and the attention to it has really increased. And I think it’s being highlighted in ways that it’s just impossible to look away from. So this specific issue is touching on, I think, a bigger kind of reckoning that faculty are having on an individual basis, and as institutions. I see a lot more individual instructors really asking, “Wait, am I being inclusive?” The question is way more in people’s minds than I think it’s ever been, in my experience.

Rebecca: Related to that is the idea of accessibility too. With so much delivery in digital formats, the topic of digital accessibility is becoming much more prevalent in the forefront of faculty’s minds, whether they want it to be or not, it becomes something that everyone’s becoming more aware of. This same kind of software also imposes a lot of accessibility issues and barriers for students with disabilities, because a lot of them are not compatible with assistive technology and aren’t built to Web Content Accessibility Guidelines, essentially.

Jessamyn: And related to that, students with anxiety issues, who are struggling with mental health issues… the high-stress, high-stakes examination, in any format, is a challenge. But add to that the technology aspect of it, you are looking at assessment mechanisms that really isn’t being accessible and inclusive, it would not allow all your students to show you what they know.

John K.: One concern that I have about proctoring services is that faculty may see it as a simple solution that will allow them to use tests that they’ve created in the past. Many people have created very elaborate test banks in Blackboard and other places and then they expect that those questions can now be used, if they’re used in a proctored environment, not realizing that most of those questions have already been distributed to multiple sites out there and students would often have access to them, anyway. So I think that proctored systems can provide instructors with a false sense of security and as John mentioned earlier, they can be pretty easily defeated as long as students have devices that will allow them, for example, to do screen shares in the background underneath the proctoring service or perhaps have multiple devices where they can be looking up answers or using some other mechanism that won’t always be easily detected by the proctoring service.

Jessamyn: That’s a good point, and I know John Locke has addressed that issue. I mean, you don’t drill in on it, but when you’re talking to faculty, you often say, “And by the way, this is not a magic bullet, even if you go through all the trouble of setting it up.”

John L.: The idea that somehow having someone else proctor your exam is going to save you time…. That’s not how it works. These proctoring systems just flag potential incidents. You still have to go through and you decide whether or not those are warranted as cheating or if they’re just someone sneezed. So, between setting up the exam and then reviewing the flags, looking for false flags, I don’t know if it saves anybody any time.

Rebecca: I’m team workload reduction.

Jessamyn: Yes.

John L.: Yeah. [LAUGHTER]

Rebecca: So, what do we say to faculty who ask about replicating those high-stakes testing environments in their online environments.

John L.: I say: “Why?” I think that what would be more appropriate is to simulate the environment that somebody needs to perform in where they’ve acquired the knowledge in order to accomplish that performance. For instance, I taught a computer applications course years ago, and for the final exam…. I did have a final exam… but, I told them, “What I hope you get out of this class is to learn how to learn how to use software. So if you haven’t already learned how to learn to use software, now’s your chance. And when you’re out in the real world, you will have the software manual, you will have the person in the cubicle next to you, the only thing you won’t have is me. So, unless you have a question about a specific question on the test, don’t talk to me, as far as anything else that gets you to accomplish the goal, go for it.” If you’re studying to be an ER doctor, perhaps you do need to have the pharmaceutical manuals memorized page by page. [LAUGHTER] But most of us aren’t working in that kind of stressful environment. So, there are better ways, maybe project-oriented ways, to assess that that learning has taken place, that those skills have been received or learned and received.

Jessamyn: I try to assume best intentions on the part of all faculty. And I know that many of my colleagues who expressed that sentiment exactly, like “How can I make sure they’re not cheating?” …they’re not saying that because they’re evil, like “Mwah, hah hah hah, those bad students…” No, they really are concerned about student learning. So, what John and I did was really to frame this as an invitation to faculty, an invitation to think creatively about assessment, authentic assessment, to really be able to measure student learning, but maybe also rethink what you thought and assumed about assessment. And here’s a big bonus, maybe grading it could be less painful. If you are trying something new, something that’s a little bit more creative, that might help you as well in your end. So, that’s been how we’ve been addressing it here at Plattsburgh.

Rebecca: What are some ways to do that assessment, maybe in a class that doesn’t work well for project-based learning. Maybe it’s a bigger section class, or maybe it’s more foundational information that doesn’t lend itself as easily to project-based learning. What are some alternatives?

Jessamyn: There’s always small, lower stakes, regular quizzes. So instead of one big, huge exam, having smaller quizzes along the way. That’s just one off the top of my head… an easy one. John?

John L.: Yeah, well, especially in this environment, discussion forums are really, I think, underutilized. There’s no reason that you can’t build a rubric around a discussion forum and spell out your expectations to students and then hold them to them and grade according to those. Again, it’s taking the student higher up that Bloom’s taxonomy ladder than just memorizing and regurgitating information. It’s causing them to react to other people’s comments within the discussion forum, to assimilate the knowledge that they’ve already accumulated, and to create new and different responses based on that immediate situation. And, the advantage to that for slow thinkers like me, [LAUGHTER] is that you don’t have to be quick on your feet. You’re not the student in the back of the room with his hand up saying, “Well, never mind, you covered that five minutes ago.” It’s kind of an equalizer. I wouldn’t say “Have a discussion forum as a final exam,” but it’s another part of the scaffold to assess that learning is taking place throughout the semester.

Jessamyn: I think there’s a lot of potential for open-book exams as well. In fact, I have used open-book exams for a long time. And, in large part, that is because I really wanted my students to learn, and I wanted to be able to grade an exam very rigorously. So saying, here’s a question you can answer with an open book, and, yeah, you might even talk to someone else about it. But then the final product is an essay question, or it could be a presentation, it could be a sort of annotated bibliography. There’s lots of ways it could go as an open-book exam. But then when I go to assess it, I know that you have the material in front of you. So, I am going to really drill down here, like, “Do you really understand this concept? Can you show me that you understand it?” Because I know you can look at the basic definition in the book that’s open in front of you. So, now you have to show me that you really, really get it, you have to use it, you have to apply it, whatever it is.

Rebecca: What about STEM-oriented examples? A lot of the things that we’ve talked about work really well in the humanities and the arts. How about some things that work well in math and science and other STEM fields?

Jessamyn: So, I’ve been trying to do a little reading in this area. I’ve been hearing from some faculty in this area. So, in an online lab setting, being able to complete the experiment in the correct way, in the scientific-y way… [LAUGHTER] …that could be one way to assess learning… doing something like a fact sheet. So the final product is how you’re assessing the student learning. But again, you could be measuring the application, the correct way to do XYZ in a kind of fact sheet format or a PowerPoint slide or a poster presentation.

John K.: One type of thing we sometimes recommend for people in the STEM fields is that, if they are going to use multiple choice, one way of dealing with this is to use some algorithmically generated questions so that each student gets their own version of the question. Now, the solution procedure may be the same, but for at least low-level skills, that can help to deter some academic integrity issues.

Jessamyn: Student-generated exam questions could be another way to go. If you really understand the material, you’re not just regurgitating memorized material, but if you really understand it, then you should be able to help someone else understand it. And one way you could assess that would be “What are the 10 best exam questions?” …something like that.

Rebecca: Another idea that I’ve heard from people more in the STEM areas is the idea of creating some sort of resource that explains a topic to a non expert audience. So, maybe it’s an experiment or something that you can do with kids, or just kind of generally to someone who’s not in the discipline and get them to grasp whatever it is that you’re trying to assess.

Jessamyn: Yeah.

John L.: This might be going out on a limb for a STEM environment, maybe we could call it STEAM, because there is an artistic bent to it. But, for instance, in an accounting course, if there’s a particular accounting procedure or process that students have to prove that they understand it, they could write a short story, “a day in the life of the accountant to the New York Yankees” or something… and totally fictional, but covering each step in the process that has to be accomplished. And as an instructor, I would love to read something like that rather than checking off right or wrong on a test sheet.

Jessamyn: I’m thinking too about something like following up Rebecca’s suggestion, and increasing accessibility, you could even have students creating resources like that in a variety of formats. It could be a poster, could be a podcast, could be a video, could be a live presentation… You could do something like an oral exam… something like that.

John K.: One of the things I’m doing in my small class of 60 students is having students create podcasts. Unfortunately, that doesn’t scale as well, in my class that’s closer to 300 students. So, I’d really like to do more open pedagogy projects. It’s just, in large intro classes, that’s a bit of a challenge.

Rebecca: John, you have some experience using algorithmic questions, too, as a way of assessment, right?

John K.: Algorithmic questions can work very effectively, in at least making sure that students can use the formulas appropriately, which is a basic skill in many STEM classes.

Jessamyn: What I would like to see is more faculty really having these discussions and swapping these ideas, like on a national scale. I think that the learning curve has been so high for so many instructors in so many ways. Like, not just, “I’ve never even visited the learning management system, and now I have to use it.” Not just that. But, coming to terms with the emotional aspects of teaching and trauma-informed teaching in the midst of, possibly, “I’m at home and I’m supposed to be overseeing my children’s education” or simple childcare issues. All these things are overwhelming so many instructors just day-to-day life. And then on top of that, “Oh, rethink something you’ve used forever. The thing that you relied on from day one, and that you did so well in graduate school… hey, that’s not gonna work.” That’s hard. That’s tough. So, the more sharing of ideas we have, and the more spreading of good possibilities for assessment, the better. And I sent you a list of some of those resources I’ve been providing. They are starting to be generated, especially at university teaching centers and in people’s blogs and essays and such. But, I think the more it just becomes a broad conversation about “What can we do? How can we, in this situation, assess student learning in new ways and recognizing it’s new for us, too.”

Rebecca: Bill Goffe, in our episode 154, Sharing Disciplinary Pedagogies, also offered a way to get people to collaborate across institutions on some of these kinds of things using a simple Google Sheet. So, we’re all kind of forced to be on line in some capacities now, maybe more than before, but maybe that’s also opening some doors for collaboration that haven’t been there before, either.

Jessamyn: I hope so. I mean, John Locke and I, both of our centers had not been collaborating in the past. So, spring of 2020, was like this kind of completely perfect context for us to send a message to the university, the Center for Teaching Excellence, and Technology Enhanced Learning, we work together, and because people needed us both. So, in that sense, I won’t say silver lining, there’s no such thing right now, but it was a unique opportunity for these two very small centers on campus to collaborate.

John L.: Yeah, in fact, I’ve accidentally come up with a tagline that is starting to appear at the bottom of my emails to faculty. And that is, “you are not alone.” They never were, but it’s much more important for them to realize. In fact, I was working with a professor last night who was having some difficulty in the learning management system. And about 10 o’clock, I sent him what I thought was probably the solution. And I didn’t hear back. So, this morning, I sent him an email and said, you know, “How did it work out?” And his response was, “I’m sorry, I haven’t even gotten to it yet. I’m sorry.” And I said, “No, you don’t have to apologize to me, I just want you to know that you’re not alone, that I’m trying to help you. And I’m not going to let go until I know your problem is solved.” And that sort of community approach to learning in general, and what we’re all going through, I think is helpful. If you know that I know I’m struggling with this I’ll bet someone else is too and, maybe between us, we can figure it out. If more people can adopt that thought and not feel that they’re infringing on someone else’s time, I think we’ll all get through this to whatever the other end looks like.

Jessamyn: That was one of the first things that John Locke has said to faculty who wanted to use this remote proctoring system is “Don’t make your life harder than it has to be.” All the student issues aside, and equity and trust and accessibility, but it’s such a pain in the ass. It really is hard to use. And I’m not just talking to the student end is terrible, but from the instructor end. It’s such a pain to set up and he shared with me, sometimes someone will approach him, “Can I set this up,” he said “Okay, but you have to do bla bla bla bla bla, then this and this…” and they’re like, “uh, maybe I’ll rethink this.” LAUGHTER] I mean, let’s try to make our teaching a little bit more joyful, if we can. Let’s try to make it a little bit more creative, for our sake, if nothing else,

John K.: It can be a lot more fun listening to podcasts students create, listening to their videos that they create, looking at documents they create, or infographics and other things, than it is reading a pile of exams, or writing up multiple choice exams.

Jessamyn: For students, too. Conveying their knowledge in a different way. It’s so good for their brain. That’s why I’m always reassuring students, when I’m asking them to do non- traditional assessments, which I mostly use (even before all this). Our students are very traditional in many ways, and they get really nervous when I say, “Okay, so you’re gonna write a short story, you’re gonna do a poster.” And they say: “Wait, what? I’ve never done that before.” Or “ I don’t know, I don’t know if I can do that successfully.” And I’m constantly telling them, “This is you conveying your learning, your skills, your knowledge in a new way, and it feels challenging, but you could do it and it’s great for your brain. It’s like calisthenics for your brain. You’re presenting what you know, just like you would in a traditional research paper or a traditional exam, but it’s in a different format, and that’s great for your thinking in all ways.”

John K.: We always end with the question, what’s next?

John L.: What’s next? I’m waiting for that chip to be implanted in my head so that I won’t have to show you my assessment, you’ll just be able to download it. [LAUGHTER]

Jessamyn: John, what is your next book project?

John L.: My next book project… I’m writing a novel that’s called “Defending Eldorado” and it takes place in South America, about 50 years after Columbus, where a bunch of colonial powers are trying to find Eldorado and the native South, Central and North Americans are doing their best to make sure they don’t find it. And since we never did, obviously, they were successful. Spoiler alert. [LAUGHTER]

John K.: You mentioned that you had just completed a book. What was your most recent book about?

John L.: Ah, my most recent book was actually the prequel to the current book, a nd that was about a group of disillusioned European scholars who left the Academy. They were humanists, they left the academy because it was being run by scholastics. And they decided to find Thomas More’s Utopia, which leads them to the New World, and hilarity ensues. Not really, but… [LAUGHTER]

Rebecca: How about you, Jessamyn?

Jessamyn: I’m headed, coming up, very shortly, I think everybody here is familiar with it, the SUNY Faculty Developers Conference, it’s going to be online and I’m doing a poster there about a series of events that John Locke and I hosted over the spring for faculty. So, that’s coming up next month. I’ve got some speaking things coming up. I’m really excited to be speaking at the Lilly Online Conference in November, and I am reading chapter submissions for an anthology project that’s contracted with West Virginia University Press in their Teaching and Learning Series. It is an anthology of insights into effective teaching and learning from women, marginalized, and underrepresented faculty. I have some fantastic submissions… so many good ones. So, that’s been a really great thing I’ve been working on right now. It’s fun.

Rebecca: Well, thank you both for joining us…

Jessamyn: Thank you.

Rebecca: …and we look forward to your future work, for sure.

John L.: All right, thank you.

John: It’s great talking to both of you.

Jessamyn: Nice to see you both. Hang in there, SUNY Oswego.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Brittany Jones and Savannah Norton.

[MUSIC]