The growing capabilities of generative AI platforms have made it increasingly difficult for faculty to reliably distinguish between student work and AI-generated output. In this episode, David Wiley joins us to discuss the possibility of using random audits to promote academic integrity in a scalable manner.
David has an extensive record as an innovator, entrepreneur, and leader in open educational materials beginning with the Open Content Project in 1998, continuing with his work as Director of Educational Licenses for Creative Commons, a co-founder of several education-related organizations, including Lumen Learning, where he also served as the Chief Academic Officer from 2012 to 2025. He is the recipient of numerous awards, including an NSF Career grant. David is currently an Associate Professor at Marshall University where he teaches courses in Entrepreneurship and Management Information Systems. Much of his recent work has been on the intersection of generative AI, open education, entrepreneurship, instructional design, and student success.
Show Notes
- Wiley, David (2026). Random Audits as a Scalable Deterrent to Cheating: Using Game Theory to Design Fair and Effective Academic Integrity Systems for the AI Era. Working paper.
- Akerlof, George. (1970). “The market for lemons: Quality Uncertainty and the Market Mechanism” The Quarterly Journal of Economics. Vol. 84. No. 3. August.
- Wiley, David (2025). Using ChatGPT’s Agent Mode to Autonomously Complete Online Homework (video)
Transcript
John: The growing capabilities of generative AI platforms have made it increasingly difficult for faculty to reliably distinguish between student work and AI-generated output. In this episode, we discuss the possibility of using random audits to promote academic integrity in a scalable manner.
[MUSIC]
John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.
Rebecca: This podcast series is hosted by John Kane, an economist…
John: …and Rebecca Mushtare, a graphic designer…
Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.
[MUSIC]
Rebecca: Our guest today is David Wiley. David has an extensive record as an innovator, entrepreneur, and leader in open educational materials beginning with the Open Content Project in 1998, continuing with his work as Director of Educational Licenses for Creative Commons, a co-founder of several education-related organizations, including Lumen Learning, where he also served as the Chief Academic Officer from 2012 to 2025. He is the recipient of numerous awards, including an NSF Career grant. David is currently an Associate Professor at Marshall University where he teaches courses in Entrepreneurship and Management Information Systems. Much of his recent work has been on the intersection of generative AI, open education, entrepreneurship, instructional design, and student success. Welcome David.
David: Thank you. [LAUGHTER] I appreciate the invitation to be here.
John: Well, we’re very glad you accepted. Today’s teas are:.. David, are you drinking any tea with us today?
David: I’m a water guy. I’ve got a giant glass of water here. I will be drinking throughout. [LAUGHTER]
John: Hydration is healthy. And Rebecca?
Rebecca: I’ve got English Tea Time today, John.
John: And I have a black raspberry green tea today.
Rebecca: Oh, that sounds nice. I’m sure it smells nice, and I probably wouldn’t like the taste of it. [LAUGHTER]
John: It’s delicious, and it smells nice too.
Rebecca: We have an ongoing discussion of that. [LAUGHTER] David, we invited you here today to discuss your March 9, 2026, working paper entitled “Random Audits as a Scalable Deterrent to Cheating, Using Game Theory to Design Fair and Effective Academic Integrity Systems for the AI Era.” You begin this paper by noting that there has been a long standing arms race between students engaged in cheating and faculty attempting to detect and deter cheating through the use of surveillance technology such as remote proctoring AI detection software and lockdown browsers. Is this an arms race that faculty can win given [LAUGHTER] the vast amounts of funding going into the development of generative AI platforms?
David: I don’t know that it was an arms race they could win before generative AI became a thing. It seems like there’s boundless, infinite creativity and energy on the part of students in this regard, in particular. We tend to be little older, a little less connected to everything that’s going on currently, and the availability of new techniques and new tools, I don’t see any end to this arms race. If we’re trying to fight it, technology versus technology specifically, there’s nothing we’re ever going to create as instructors that’s going to compete, as you say, with these multi-billion dollar generative AI tools that are being created.
John: What you suggest is that we skip the arms race entirely by perhaps using an audit system that may alter students’ incentives. Could you give our listeners an overview of how an audit system might work?
David: Yeah, it might be easiest to begin explaining by reference to some examples that people already know. We often make progress on hard problems by realizing that a different framing of the problem can help us see it from another perspective. This is a problem that I’ve been interested in for a while. Other people have been more interested in it than me, I acknowledge that. This isn’t my area of expertise by any stretch. I was just trying to abstract this problem up to the highest level I possibly could, and think about it from that perspective, and I would say that from that very high level, there’s some largish number of people that we need to be honest, but the number of them is so large and the circumstances are such that we can’t directly observe all of them. If you frame it that way, and then start to think about, what are other domains where that same problem exists, and how do they approach it, then some other examples start to present themselves. Doing taxes is the first one that came to mind. The IRS has an interest in all of us being honest, but obviously can’t observe all of our documentation and all of the work we do as we prepare our taxes and submit. Once you get to that first example, then others kind of start busting through. You think about anti-doping in sports. We can’t watch everything that every athlete does all the time, but we need them to be honest about the fact that they’re not using substances that we don’t want them to use. Nuclear proliferation [LAUGHTER] would be another… food safety, we can’t be in every warehouse producing every kind of food, the FDA can’t all the time. We need them to be honest that they’re following safety guidelines and meeting standards. Once you abstract the problem up one level, you’re able to see that same problem manifest in other domains. And of course, the way that the problem is addressed in all those other domains is through some kind of audit procedure, ostensibly random audits that have pretty significant penalties associated with them. And the idea is that the penalty is so dramatic that it removes the incentive to be dishonest in the first place. As you said, rather than trying to continue to compete in the arms race with “you develop this technique for cheating, and I’ll develop this one for countering it, and we’ll just keep doing that over and over again,” what if we just step to the side and said, “Let’s just try to remove the incentives for cheating altogether and kind of come at this a different way.” That was the originating thought behind the paper.
Rebecca: So that’s like the abstract, the big problem. And I think when you start thinking about it with these other examples, it seems like it’s not such a big deal, not such a big problem. It’s solvable. Can you talk a little bit about what that might actually look like in a classroom setting? What would an audit system look like in terms of assessments in a class?
David: I didn’t go into this very much in the paper and said explicitly in the paper that I think that there’s a second set of work that needs to be done. If people believe that this higher level framework is interesting and appealing, there’s a lot of detailed work to be done then in describing exactly what the audit would look like. And you can imagine it looking very different from one discipline to another, and looking very different from one kind of work that was done to another kind of work. So if I’ve given an exam, and this was a multiple-choice exam, say, with 40 or 50 questions, you’re selected for an audit, then what that might look like is I might take some subset of the questions that you got right on the exam, and I might watch you retake some subset of the exam over again, because I don’t want to spend another hour or hour and a half with you doing this. I want to spend 15 or 20 minutes doing it, and rather than randomly selecting questions from the exam, because maybe you missed some of them on the exam, it would behoove me to choose the ones that you’d gotten right on the exam, and we could look and see if you’re able to get the correct answers again. Or if I’ve had you write an essay, I might pull out five to 10 vocabulary words from that essay that seem kind of interesting. I might ask you just to describe to me what those vocabulary words mean to you. Give me an overview of the argument that you created in the paper. It would take a different format depending on what kind of work the student had done, which of course is related to the discipline of the course. But the idea is that you’d want these to be doable in a pretty brief period of time, 15 or 20 minutes, because you’re going to need to do a number of them, and you would only do them for major assignments. I wouldn’t do this for every 10 or 15 point homework assignment in a course. I would do it for a 100-point midterm, 100-point final, or 100-point project. I think it would need to be used pretty judiciously, because it is going to take a fair amount of time.
John: A lot of people recently have been suggesting a movement towards oral exams, and that’s a much more extreme form of this. And what I really liked about your approach is that it is an oral exam, but it’s one given only to a small subset of the students, and it’s a shorter exam designed to deter cheating, rather than to completely eliminate any possibility for cheating. And it seems like this is much more scalable than many other approaches that people have suggested. One of the things you mentioned is the Ellsberg paradox, which suggests that people are more cautious when they’re facing uncertainty, when they don’t know the probabilities, for example, of being audited, then when they know the risk, when they know what the probabilities are. How can instructors use this paradox to also deter cheating, or to help deter cheating?
David: Yeah, I think there are a couple of examples from behavioral economics that we could leverage to our benefit here as the instructors, but I think this is one. There is a interactive simulator that you can play with that accompanies the paper, where you can turn the knobs up and down on how many students there are and how many points there are in the course, tell you things like what percentage of people you need to audit and how big the penalty needs to be in order for the game theory to all work out, for the incentive to kind of evaporate, to cheat. You need to know all those things as an instructor, [LAUGHTER] but what you need to say to the class is, some of you are going to get tapped on the shoulder after the midterm, and we’re going to do this kind of supplementary learning review together. But I think it’s just as easy as avoiding saying “it’s going to be exactly 15% of you,” and then I think it’s important to come back afterwards and say I enjoyed the conversations that I had with many of you in our supplemental learning review. Make sure that people know that it’s happening, that it’s occurring. Hopefully, some of the people who are selected and who do well will come out the other side and tell their friends, “hey, you know, I got chosen for one of these.” It’s important that the word gets around and that people believe that they really are happening. If the idea of the audit as something that’s really going to occur loses its credibility, then of course, it loses all of its value as a deterrent.
Rebecca: So speaking of the idea of credibility, how do we really make sure that a system like this maintains its credibility, other than being careful about how we present it to the class.
David: Say a little more about that.
Rebecca: If students say, “Oh, I was tapped on the shoulder,” then it seems like whatever [LAUGHTER] you were tapped on the shoulder to discuss, or whatever that supplemental learning instruction was would need to seem to be like a good follow up. And that people didn’t feel like it was threatening in some way. If someone has test anxiety or something like that, it’s set up in a way that feels trustworthy and safe and that it’s fair.
David: Being fair is extremely important. Let me come back to that in a second. First, I would say, I’m not a person who gets upset easily. Let’s just say, a time that I was very upset, that I recall very clearly, was as a graduate student about halfway through my first course on assessment design and learning how to create a measurement of learning that is going to have the psychometric properties that you want it to have. It’s going to be valid, it’s going to be reliable. And realizing that my entire life had turned on decisions that had been made about what I had and hadn’t learned by teachers in high school and instructors in my undergraduate career that had no idea what they were doing as they designed these assessments [LAUGHTER] , they had taken multiple choice exams when they were a student, so they created them when they became a professor, and, oh my gosh, I got so lit up about that. I promise we’re coming back around to your question. But one of the interesting phenomena that happened with the move to online learning in the 2000s as that really started to take off, was people started making claims about the degree of quality and rigor and the standards that a course would have to meet in order to be offered online. And it was always hysterical to me that if you had turned around and tried to apply any of those criteria to the face-to-face instruction that was happening, very little of the face-to-face instruction would have passed muster. So there’s this weird kind of double standard that we’re accepting of things being, I’m going to say poorly done, that’s stronger than I mean, but we’re accepting of things being poorly done as long as they’re poorly done the way they’ve always been poorly done. But if we’re going to do anything different, it has to meet this 18 times higher bar. The reason I bring that up is that I think it’s the same person who’s going to design the assessment for the supplemental learning measurement as designed the original assessment. And so the idea that it would be somehow unfair or tricky or difficult or something like that, it will probably still be kind of poorly designed, just like the original assessment that was taken maybe was because maybe that person did their PhD in something other than education and has never had a class on what valid, reliable assessment design looks like. I don’t want to get in a place where we try to hold this secondary assessment to a 10x higher bar than we hold the primary assessment. I think as long as the secondary assessment looks very much like the primary assessment. That’s really what we want. And I mentioned in the paper that you wouldn’t want to put yourself in a situation where maybe your primary assessment, the exam that everyone does, is measuring learning outcomes at very low Bloom levels, like remember and understand. And then you would turn around on the secondary assessment, and all the questions would be apply and synthesize, and then you say, “Oh my gosh, you failed the supplemental exam.” Well, it’s broadly on the same topic, but it’s not measuring the same learning outcomes that the original assignment did. That’s why I think it’s nice in cases where you have some kind of exam with multiple questions, whether they’re short answer or multiple choice or whatever, to take a subset of those questions that the student did well on, that’s ideal. If you can’t do that, then you can take parallel forms of them, where, if it’s a word problem, you change the names of the people and the number of apples, the same question, just changing some of the details about it. But it’s very important that that supplemental assessment be on the same topic, at the same level of difficulty, similar in every way that it can be to the primary one. There’s no assessment strategy you could take to scale across higher ed if it really depended on all those assessments being valid.
John: To reiterate your point, it’s not just a few people who haven’t had much training in assessment and reliability and validity. It’s probably 95% or more of those teaching in college, I would suspect. It is a major issue, but that’s not something we can all deal with here, other than doing our best to help everyone become better at these things. [LAUGHTER] Well, going back to that point, about not sharing, necessarily, the proportion of students who are going to go through the second stage, because when students don’t know the probabilities, they’re more likely to worry about the downside of that, about the possibility that they may be caught. There’s another behavioral economic principle known as loss aversion, which is when people are faced with a choice between a gain and a loss, a loss of equal value to a gain is weighted much more heavily. There’s a lot of research suggesting that people do pay much more attention to potential losses than they do to potential gains. And I think this sort of approach is one where that loss is the main cost that students potentially face, and that may also amplify the effect.
David: I agree completely. As I worked on this idea, the loss aversion piece of it didn’t occur to me initially. Once I started doing some of the math and realizing the penalty was going to have to be bigger than 100 points to make the math math, as people say these days, that it was going to need to be 50 points more, or 100 points more. You were going to need a zero on this assessment, plus another 50 point penalty against what? …just against your overall grade. At some point, looking at that, I realized, “Oh my gosh, and people feel losses about twice as strongly as they feel gains of the same size.” That’s another bit of BE working in our favor there, in terms of, again, just trying to not fight the battle of, you’re going to cheat and I’m going to catch you. I’m just going to take away the incentive for you to cheat in the first place, so that we don’t have to fight that fight later.
Rebecca: I can imagine that introducing a system like this into a class would be challenging, like any type of different grading mechanism [LAUGHTER] where students are used to a particular kind of assessment or particular way of teaching and learning that may be counter to [LAUGHTER] good practices, but we want to introduce something new to them, we have to be really careful and take the time to explain the why. I imagine that would be the case here too, is really taking the time to explain why there’s an audit system and what the purpose of that is, would be important to getting the students to buy in.
David: Absolutely, I think so. I’ve had some preliminary conversations with folks in the Office of Academic Integrity at my institution to talk about, “Does this seem like it’s compatible with the way things work, and at what point do I refer someone to your office as having violated their academic integrity,” as opposed to “I’m just dealing with it internally, with penalties that are clearly outlined in the syllabus.” And it’s not me saying, “Ooh, I have vibes that you’re cheating,” or “I read this paper and it didn’t sound like you,” so I’m going to accuse you of something, but it’s just a repeated measures. You took one test, you did very, very well. We turned around and did a second measure of the same thing, and it was really different, and that’s a problem. So I think laying all that out and being very honest about the fact that I don’t want to do all the work it’s going to take me to do to try to catch you when you cheat, it’s just too much time. It’s too much effort. If I’ve got a class of 50 students, some percentage of them score a C or better, which creates the pool of people that I’m going to draw from to do these follow up learning measures, if I have to do five of them on every major assessment, but it actually eliminates the problem, that’s much less time spent than I would have spent on all the other things that you do. It definitely will take some time. And now, as I’m answering that question, I realized I’ve lost the thread of what the question was. I’ve meandered a bit from what you asked me there. I apologize.
Rebecca: I mean, I was just really thinking about faculty who introduce concepts like ungrading that are just really unfamiliar, because it’s very different from the ways that they’re expecting the assessment or getting caught cheating… like what any of that looks like. So for making it look different, then it really requires some explanation up front.
David: This is something that, if you don’t use a syllabus quiz, you would want a syllabus quiz where people have to go through and answer four or five questions about “Yes, I looked at it, yes, I understand I’ll lose all the points on that assignment.”” Well, and how many additional points will you lose off of your overall grade? Will it be 25, 50. 75, 100? It’ll be 100 points more. You’d really want to require them to go through some exercise where they have to read the syllabus, they have to understand, they have to demonstrate that they understand. You’d want to go over it verbally, either in front of the classroom or on a video if you’re teaching online. It’s going to take several explanations. And then I would raise the issue again just before any assessment where it applied. So everybody remembers, back in September, we talked about this, and now here we are at the big midterm, and this applies. So let me explain again what that means. Some number of you are going to take a secondary assessment after we take this primary one, and if there’s a big difference between how you do on the secondary one and the first one, there’s a penalty, and this is what the penalty is like. And the reason I’m doing this is to remove all incentive for you to cheat at all, because the odds of you passing the class if you cheat and are caught here, are basically zero. Maybe if you got 100% on every other assignment, you could squeak out a C- that’ll let you count this course toward your graduation. It’s just too dangerous, so don’t do it. You’re going to get caught.
John: And maybe even a reminder of that at the top of the assessment might not be a bad strategy, too.[LAUGHTER] So it’s there explicitly in case students ignored all the information up to that point, which has been known to happen once or twice.
David: Actually, that’s a really interesting point. Maybe the first two questions on the assessment should be, is there some chance that I’ll be selected for a secondary experience, and if there’s a radical difference between my score here and there, in addition to getting a zero on this, how many other points will I lose? Now, I wouldn’t want to put them in a state of panic that’s going to prevent them from performing well on the assessment, but you’re right, the proximity of the reminder to the task really ends up being important, doesn’t it?
Rebecca: If you really don’t know this, you should just let me know now. [LAUGHTER]
John: One of the other things you talk about, though, is a concern for false positives on this second-stage assessment, because students may be nervous, they may be having a bad day, there may be some anxiety or language issues or some other issue that interferes with their performance. So you suggest a few ways of perhaps addressing the risk of false positives. Can you talk a little bit about those?
David: Yeah, well, we’ve talked a little bit about one of them, which actually may not be in the paper, that is just been part of my thinking is that’s evolving, and that is using subsets of the same work that you had students do on the primary measure on the secondary measure, so that you remove any, what we might call construct irrelevant noise from that second measure. We want them to be the same across there. But if a student performed really poorly on the supplemental measure, they might come back and say a couple of things: “I was having a really bad day.” “My grandmother died.” “My cat died.” “This happened, this other thing happened, and I was just out of it.” Then I’d be inclined to say, “go home, sleep it off, eat a decent breakfast, and let’s try again tomorrow.” Now, if there’s something structural about the assessment that makes it difficult for them to engage with it, a) I would have expected that to show up the first time they took the assessment, b) any kind of accommodation that they require for anything that’s going on with them and their learning, we should have provided the first time around, and we should definitely provide the second time around. I think it’s important that there’s some kind of release valve for somebody who just wants to bang the pulpit and insist, “no, I do actually know this. There’s something wrong with the supplemental measure that you did” to say, “heck, let’s do it a third time. But this is the last one. With repeated measures. If I get two of one outcome and one out of the other, I know which one I’m inclined to believe now,” and when they’re quite different than that’s kind of concerning. I think the primary method is having there be some appeal process where they can take a second supplementary learning evaluation and maybe modifying the mode or the method of that. If there is something about the first one that if they’re English as a second language learner, and there’s a lot of writing on this version of it, then maybe there’s some alternate form that I can provide them that’s going to make it easier for them to show me what they really know, because the point here isn’t to catch anyone. The point is to get them to demonstrate what they actually know. So it’s not about gotchas. It’s about accurately measuring what students know and are able to do. So if there’s something about that secondary assessment that made it so that that measure was inaccurate, then we should take another swing at, but it’s not turtles all the way down, right? It’s not the 15th time that we’re coming around and trying this thing again. There has to be a stopping point.
Rebecca: Some students might not be concerned with maximizing [LAUGHTER] their grades. They might not be aiming for an A, perhaps they’re using AI to meet those minimum passing standards. Students say things like, “Cs, get you degrees.” If this is a common phenomenon, which I’m pretty sure it is, how could the audit process be modified to deter kind of that C level cheating?
David: One of the differences between the first draft of the paper and the draft that was posted has to do with this issue, and that is that I was trying to think about how to make the potential pool of people that might be drawn, that might be audited, smaller. And it occurred to me that if you failed the exam, there’s no need for me to audit you, because you failed the exam. Depending on the institution that you’re at and the discipline and the major, sometimes you need a C or better to count that course toward graduation requirements, sometimes just anything that passes is okay, but I think that as an instructor, you’d need to be kind of attuned to that, and that the floor, in other words, the lowest score with which someone would be eligible for the audit, needs to be far enough down that it picks up all the people in that category. So if it’s A, B, or C counts as satisfactory academic progress, then everybody who gets a C needs to be in the audit pool too, and they need to be drawn out, and they need to have the supplementary experience. If they knew it so poorly that even using AI, they could only pull out a C, then I think that’s going to show up, now if they’re being super clever and smart and trying to calibrate exactly their AI use so that they only get a C. Actually, in that case, they’re only cheating themselves. I mean, if they’re going to use AI, they should have just gone ahead and got an A, using it in a clever way to get a C doesn’t help you, because if you get pulled into the audit and it turns out you don’t know the material, you may as well have rolled the dice on getting an A than getting a C.
Rebecca: How to play the game well.
David: Well, this is part of why you don’t make all the rules explicit, right? It’s 12 and a half percent of you that are going to be audited every time, or something like that. I think to the extent that the rules are unclear, it’s harder to play the game.
John: Going back a little bit to addressing those false positives, you also mentioned perhaps having some type of an audit bonus or a bonus for those who are audited and passed successfully. Can you talk a little bit about why you might use that approach?
David: Rereading the paper before we got together today, I can see me disagreeing with myself a little bit in the paper [LAUGHTER] on this topic, which is why it’s just a working paper still. My initial inclination there was that if you’re going to get pulled out and have to schedule time and be at a place at a time to meet with me for 15 or 20 minutes to do something, you probably deserve something for doing that, something other than “attaboy” or “attagirl.” So introducing a couple of points that you could think of as extra credit or as bonus points, or as thanks for being part of this process, was the way I originally thought about that. It turns out that that bonus also helps the math work out in terms of the expected amount of points somebody could earn if there’s a chance of a false positive. But then if you don’t get a false positive, and you are able to show what you really knew, then you do get a couple of extra bonus points, like making that all balanced out, that it turns out to work out nicely from that perspective as well. Later in the paper, you can see me kind of arguing with myself about whether I want to call this extra credit or bonus points or, like, “I put you through this whole thing, guys, I’m going to throw you a bone. Here’s a couple of points for participating in this process,” versus saying “you took this much smaller second assessment, and since it was smaller, I’m going to give you a smaller amount of extra points for passing it, because you get points for passing the first one, get more points for passing the second one.” I’m still not exactly sure how to frame the language around it, whether to think about it as extra credit, or look, you have a chance to do a second assessment, and you’re going to earn more points on that, and those points are also going to count toward your grade, and other people aren’t going to get the opportunity to take that second one and not have the opportunity to earn those extra points. So you can think about it as being earned or being extra, or however you want to think about it, you can really only understand by putting this in front of students and working through it a time or three. There’s got to be some way to talk about it, though.
Rebecca: Like thinking through a student lens. “Ooh, I want to get audited so I can get some extra points.” [LAUGHTER]
John: If only the IRS worked that way.
Rebecca: I was just thinking. But I’m like in all these other contexts that doesn’t play out.
David: That does not work that way in any of the other contexts. And maybe this is just me being a softie, especially in an asynchronous, online course, if I’m going to require you to make an appointment and show up someplace at a certain time, I think you deserve something.
Rebecca: I think that feeds into the question of just in what ways is the system equitable, if we think about the student populations and thinking about your sampling and different strategies that might be employed. How do we make sure that a process like this remains equitable?
David: I think it being truly random is important to being equitable. And there’s a paragraph or two in the paper, some people who did poorly on the first two or three assessments, and now they suddenly do really well on this one, maybe we might weight their potential to be drawn out of the pool a little more heavily. You could certainly do something like that, but it seems like the most fair way to do it is just to do it completely randomly. And I think part of the equity argument here is that it’s really only fair if the grade that you earn is an accurate reflection of your learning. And if you work and study and push and strive and you end up with an 89 and then somebody else who does none of that and cheats their way through ends up with a 93, that, to me, is like the definition of being inequitable. Part of the equity of this is who is selected to participate, but there’s also an equitable outcome of does the signal of the grade truly reflect your learning in a way that is fair? Everybody’s grade really is an accurate reflection of their learning, not some people’s grades are accurate reflections, and other people’s grades are super over inflated because they were cheating and we weren’t able to catch them. I think about the equity on those two levels.
Rebecca: I think the transparency that we talked about earlier too, about explaining the process and making sure students understand the process, that plays into equity as well. It’s really important for everybody to understand a system that might be really unfamiliar to them.
David: Not that I’m trying to convince you. I really would want to try to try to convince them that this is not about trying to catch you doing something. It’s a second measure, and if you did well on the first one, you’ll do well on the second one, and you’ll get some additional points for doing well on the second one. I think there’s a way that that can be explained and communicated and practiced that is fair for everybody. If it doesn’t eliminate the incentives to cheat, it can really dramatically reduce it.
John: And one of the things I was just thinking about, as we’ve been having this discussion, is a classic economics paper by George Akerlof called “The Market for Lemons,” which explains why, when you drive a car off a lot, its value drops dramatically. And the reason is, when you buy a new car, you can expect that most new cars are very high quality, but if you look at the market for cars that are a year old, what happens is, we get a very different mix. The high quality used cars are ones people are likely to keep, the lemons that they are having a lot of problems with are the ones that tend to be offered for sale. In the market for used cars, you don’t necessarily know whether this used car would be a lemon or not, and that’s essentially the sort of system we have with grades in the presence of AI. The whole value of a degree and of the grades in classes can very well be lowered if employers and future instructors at other institutions observe that students have not mastered the skills that their grades suggest that they have, and that can devalue education for all the students, including the ones who actually did master the skills we hope that they have. Having more accurate signals is a really useful way of addressing this, and that’s one of the things I really like about this approach.
David: It seems like now, more than any time that I’ve been alive, for sure, that the number of questions about the value of education are already at an all-time high, confidence is at an all-time low, perceived usefulness is at an all-time low. AI just has the potential to make that situation even worse if we don’t figure out some way to really improve the consistency of if you came out of that program with a B average, I know what that means in terms of skills and knowledge and attitudes and things that you’re going to have, but when those signals don’t mean anything, then people stop relying on them. And, I tell you what, you want to see, the bottom really fall out of higher education, we just need employers to stop requiring degrees as a condition of employment. If a degree isn’t the gateway to good employment, who’s going to keep showing up paying what they pay now to participate in this process. We’ve got to do better.
Rebecca: I think the more we share with students the concern that we have about their learning and why we might put measures in place to make sure that their degree does mean something, there’s a lot of motivation there to focus on learning too, having those setup conversations like motivating in a lot of different ways, motivating because you’re explaining why you want to make sure their degree has value, that that demonstrates care, and that’s good relationship building. But also I care that you learn. I think those are things that can be really valuable, and some of that’s baked into the system without explicitly using that language. There’s a piece of care that’s present.
David: I indicated earlier that it’s possible that I’m a softie, but I am one of those people that tells my students on the first day of class that I don’t know you yet, but I’m here because I love you and I care about you, and I think this whole experience can be one that changes your life, improves your life, the life of your family for generations, there’s real reasons that we’re here. I could be making a lot more money somewhere else, but we’re here to do what we’re here to do, and I don’t think you can talk about that enough. I think it is so easy to reduce it to gameplay. I need this many points, and I get that, and what’s the cheapest, fastest, easiest way to get those points? If we don’t call them back to help them think again about why are they here. Tell them why we’re here. Explain what we’re trying to do together. If we don’t do that, we certainly can’t expect them to be responsible for providing all that context and motivation for themselves. That’s part of what we get paid to do.
John: One of the really nice things about your approach is that AI systems are getting better and better and harder to detect. I remember seeing not too long ago a video you had posted of an agentic system going in and completing all the work in a simulated course without any intervention other than providing the login ID, the password, and the URL. And in that environment, we really need to do some deterrence, since we can’t detect it reliably any longer, and this seems like this approach is independent of how AI systems develop, is that an accurate description of your view of this?
David: It is today, and that’s the goal. I don’t know, it’s the academic in me. I expect someone is going to stand up and raise a counter argument at some point that is going to burst my balloon. The way that I look at it now, it really does feel like it steps back from the way we currently think about it, and steps to the side [LAUGHTER] and steps forward again in a way that it really doesn’t matter what AI can do. You could have it log in and do all of your assignments for you fully autonomously. If you get selected for the audit, and we are going to have a conversation about it, and I’m going to watch you do it and talk through it, then it doesn’t matter if AI wrote your essay, or if you paid someone in some other country to write your essay, or if your friend wrote your essay for you, it doesn’t matter how you cheated, because this is just repeated measures of learning. I took one measure and it showed this, and now I’m going to come back and take another sample and try to find out. So I really do believe today that it’s independent of the kind of cheating that happened. Doesn’t matter what tool you used or what approach you take, so long as I can get to you for some secondary, supplemental follow up, however you want to describe it, where we can really be knee to knee, either virtually or physically, and have an experience where there’s not an opportunity for you to cheat in that same way that you cheated the first time. Now is it possible that we could log on and it could be your video avatar that’s being powered by AI. I suppose there’s some scenarios like that, but we’re not there yet.
John: So we should be safe at least for another six months to a year. [LAUGHTER]
Rebecca: So we always wrap up by asking, what’s next?
David: I’m currently playing with some technology to help a person who would want to implement this strategy be able to implement it more easily in their class. I think it’s one thing to hear it described and for it to be in a paper. For example, if it was an LTI tool that you could add to your Canvas, your Blackboard, and it would provide all the support for you and doing the random selection and pulling the subset of questions that students have gotten correct and provided all that support to you, it’d be a lot easier to use these kinds of approaches. I think for me, the next steps on this work are keep talking to people and waiting for somebody smarter than me to show me why it’s a dumb idea. And while I wait for that to happen, try to create some tools to make it possible for people to start using this approach, because we really desperately need some way to combat, some way to slow down, change the incentives around cheating.
Rebecca: Well, thank you so much for joining us. I really appreciated the reframing and getting challenged to think about the problem in a different way.
David: Thank you. This was so fun. What a treat. What a blessing to be able to put an idea out into the world and have other people engage with it and then have a chance to talk about it. This is academia at its best.
John: Well, thank you, and thank you for all the work you’ve done over such a long period of time.
David: Thank you.
[MUSIC]
John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.
Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.
John: Editing Assistance provided by Fred Llerena.
[MUSIC]