465. Scaling a Multicampus AI Initiative

Keeping up with the rapidly expanding capabilities of generative AI platforms is challenging, especially at campuses with heavy teaching loads and limited resources. In this episode, Racheal Fest joins us to discuss a multicampus AI professional development project that addresses some of these challenges.

Racheal is a pedagogy specialist at the Faculty Center for Teaching, Learning, and Scholarship at the State University of New York at Oneonta. She also teaches writing courses in the English Department there. Racheal was a principal investigator on two SUNY multi-campus grants focused on faculty development related to AI.

Show Notes

Transcript

John: Keeping up with the rapidly expanding capabilities of generative AI platforms is challenging, especially at campuses with heavy teaching loads and limited resources. In this episode, we examine a multicampus AI professional development project that addresses some of these challenges.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Racheal Fest. Racheal is a pedagogy specialist at the Faculty Center for Teaching, Learning, and Scholarship at the State University of New York at Oneonta. She also teaches writing courses in the English Department there. Racheal was a principal investigator on two SUNY multi-campus grants focused on faculty development related to AI. Welcome back, Racheal.

Racheal: Thanks for having me.

Rebecca: Today’s teas are… Racheal, are you having tea with us today?

Racheal: Yes, I am. I love tea, and today I am just drinking my standard, which is PG Tips English Breakfast. I’m on my second cup.

Rebecca: That seems right for the day, for sure.

John: A very popular tea in England, and we often have that in the office, it’s a really good tea. My tea is spring cherry green tea.

Rebecca: Oh, nice, John! Bringing back your favorites. I have an Irish breakfast this morning, so definitely breakfasting it up with Racheal for sure. Some extra caffeine.

John: On an earlier podcast, we discussed that first multi-campus AI initiative for which you were the principal investigator. Today, we’d like to focus on the second multi-campus AI professional development. But it would probably be helpful for those who don’t remember or did not listen to the first podcast to set the stage for it a little bit because the second grant grew out of the first to some extent. Can you give us a brief overview of that initial project?

Racheal: In the 2024-2025 academic year, we worked with a team of six campuses, which included Oswego, Oneonta, where I’m located, and then we had colleagues from SUNY Alfred, Morrisville, Orange, and Schenectady joining us for Teaching with AI: A Cross-Campus Community of Practice. And in that project, we really focused on developing a set of common materials together related to critical AI literacy, and participants designed assignments through that project to help students evaluate AI outputs from a critical perspective appropriate to the range of disciplines that faculty were joining us from, and faculty earned a modest stipend for doing so. They met on campus with educational developers and faculty leaders to go through our intro to AI literacy programming, and the cross-campus element came in through small virtual groups that connected them across the six campuses.

Rebecca: Can you remind us how many faculty participated in that initial group?

Racheal: So we had six campuses, and we initially secured funding for 10 faculty per campus. We also were able to gain some additional funding from Oswego and from Oneonta, and so we had stipends for over 60 faculty. I think we ended up concluding with 66. We had some dropout for personal reasons, and some campuses had smaller cohorts to begin with, just depending on resources and availability. But yeah, we finished out with about 66.

Rebecca: Great. Can you talk a little bit about how this new project relates to that older project?

Racheal: When we scaled up the second round of IITG funding and we proposed the project to SUNY, we had more campuses involved, we had had some campuses reach out to us because we had publicized and shared the communities of practice model. And one of the aims of working together across campuses is to really pool resources and think about efficiency within the professional development that we’re offering. I think we’re lucky in SUNY that we have a lot of professional development opportunities, both through our individual campuses and through the Center for Professional Development that supports faculty across the system, and one of the both challenges and opportunities that comes with that is how we can come together to use those resources to develop support programming in a way that makes sense. And so both of these projects aimed to do some of the support programming for faculty. Basically, we wanted to plan together. We said “We’re all on our individual campuses. We’re all trying to support faculty in responding to AI.” The way faculty are engaging with the emergence of AI is very different, and for me, a real passion point for both of these grant projects, I wanted to get faculty remuneration for doing this work. AI has been a huge disruption to higher ed since ChatGPT emerged for free in November 2022, and we’ve seen all different responses, all different means of engagement, all different levels of information that faculty are coming to this problem with. And so, both the smaller grant and the larger grant aim to say, “How can we come together across the system and provide faculty with the support they need while also giving them funding?” So we, again, were giving faculty stipends to do some course redesign work. We moved from six campuses to nine, and we had funding initially for 90 faculty. We did end somewhere in the 80s. Again, we always have some attrition from folks who just have things come up, or they realize the time commitment doesn’t work for them. But we were able to offer stipends again to about 90 faculty. And one of the primary differences in this grant year, initially we had all worked together on one theme and one set of materials, that was basically designing assignments for critical AI literacy. The new project, we had educational developers from multiple schools. We had a leadership team of over 10, I think, and we decided that the best way to use our collective efforts and resources would be to offer multiple themed virtual communities of practice, and so we offered five of those.

John: And so, in that first year, people were just getting used to AI because it was only two years into the process, and it was new to everybody, but one of the reasons I think why we split up into more topics is faculty had been working with AI to some extent. Some people were coming into this new, and one of the groups that was run was very much like the first-year group, but the others were mostly for people who wanted to explore things a little deeper. What were those five groups?

Racheal: Yeah, I think that’s a great way of describing it, John. And I don’t know if we’ve said out loud yet that John was the co-lead of one of our groups. I had led the group that focused again on designing assignments for AI literacy. That was great because I was able to use the materials that we had developed again and bring them to more people. Again, that was focused on an assignment redesign. The other four groups, starting with the one John led with David Wolf, a colleague at Schenectady, focused on building chatbots and specifically in the context of study tools. I think it was. John, do you want to say a little bit more about what you did?

John: We did ask people to develop two chatbots, one of which would be a Socratic tutor that would assist students in understanding, but would not directly answer questions. And we let the other depending on what their role was, many of them did create things that were useful in teaching, but some of them were working in other areas. Some were instructional designers, some were faculty support, and so they wanted to do something a little bit different. One of the more interesting things, was someone developed an interactive simulation of a patient role, where people would interact with a chatbot that was actually stored in, I think, a dummy type of a thing, where basically the student would ask questions of the dummy and it would respond as a patient from a random selection of diseases and so forth, and report symptoms with some level of accuracy. And it was set up with a microphone and a speaker there, so it was as if they were interacting with the real patient. So that used verbal interactions. Others were set up for very specific assignments, such as analysis of a particular text or interacting with a text more deeply. There was a pretty wide variety in terms of what people did with the other chatbots.

Racheal: That’s great, John. Yeah, I was always really excited to hear about the work that you and David were doing, and I’ll share a little bit more of the other three groups that ran as communities of practice virtually. But I’ll just emphasize now that build a chatbot group that focused on some of these Socratic tools that are available. We wanted to make sure that we were offering new and deeper engagement with emerging tools. That was one of the things that faculty asked for. So, as we came up with these five different community themes, we wanted to both serve beginners, people who were still coming on board and wanting to learn about what exactly AI is. We had multiple faculty who hadn’t even looked at ChatGPT or Copilot or Gemini or whatever available tool they might have looked at, but then we also had, as John already said, many faculty who were ready to take the next step and to get some support in exploring in conversation with community members from across the system. So, I think John, your group did a wonderful job there. We had AI and accessibility, which was a group run by Luis Colon at Stony Brook, and initially with John’s colleague, Stephanie Pritchard, who ended up moving away from the university. That group focused on AI’s applications for accessibility needs and projects, and that was one of the few groups that did not create student-facing materials. Most of our groups were focused on pedagogy and instruction. Luis’s group had instruction in mind, but they were examining AI accessibility tools with an imagined audience of colleagues, other faculty. So, if they learned about a tool that could be useful to them for a classroom application, be it captioning or something else, they created deliverables that would share the information and what they had learned related to AI and accessibility. And then the final two groups that ran, we had AI and writing across the disciplines, which Matt Newcomb from New Paltz and Laura Pierie from SUNY Morrisville co-led, and finally we had redesigning assessment in the age of AI, which was led by Alexis Clifton, Deepa Deshpande and Jenny Zhang, and they focused, as the title suggests, on creating new exams, new assignments that were responsive in some ways to AI. So some faculty in that group were focused on AI-proofing assignments; others focused on integrating AI to update their assignments, so there were multiple approaches in the group, and I think that reflects all of the groups and in a common ethos that we developed together at the beginning. Not only did we want these groups to work for faculty who were bringing a range of levels of experience with AI, but AI is a very ideological topic. It depends on your disciplinary formation. It depends on your view of these tools and their emergence. And we wanted people across the spectrum of attitude toward AI to feel welcome and to be in conversation with each other as well.

Rebecca: Running communities of practice across institutions can be both exciting and challenging at the same time. Can you talk a little bit about some of the logistics of how you were able to make those communities of practice work and talk across campuses, and maybe some of the benefits and challenges that you faced in doing that?

Racheal: Yeah, there were definitely both challenges and rewards to a big project like this. I will say that I think approaching with an aggressive logistical plan [LAUGHTER] when you know that you’re going to be working both with a large leadership team that’s working online at multiple institutions is a really strong way to start. And then also we had about 15 to 20 faculty members in each of the five groups. So thinking about how to manage these schedules to make those conversations happen within each group, these were all concerns and considerations that we definitely foregrounded going into the planning. And I think it really did help that we had had the previous year’s experience. Being able to scale up and build on something we’d already done and succeeded at felt really comfortable this year. At the same time, I would say that the structure of the project really changed and the way we were working really changed. In the first year, we felt like a group of collaborators. We were educational developers and faculty from six campuses who were working primarily with campus folks in person. So we would come together. We would create the rubric for our deliverables. We would create the presentation that we were going to give to faculty each month to help them work toward those deliverables, and we had a lot of time to share, troubleshoot, bring our ideas together. The second year as the project lead, it was more of a logistical position. I was really more there to just keep us on track, keep things moving, and that was exciting too, because it meant that our leadership team members could each develop content the way they saw fit for their group and the way they saw fit for the theme that they had developed based on their expertise and interests. So there was more freedom there, and I think that was really positive in terms of drawing upon the expertise at different institutions to serve faculty on different campuses.

John: One of the nice features of it, from my perspective, is that if we tried to do a year-long development project on a specific topic, it would have been unlikely to have 10 to 20 people who would have persisted for a year. But when you were dealing with nine different campuses, there were people who were interested in spending a year on developing those specific tools and skills, and so that was a really nice process. The fact that we had multi-campus participation made it really valuable in a way that it would be really hard to do in a small campus or on a campus where you just don’t have the same level of focused interest. By providing a number of choices, faculty members from each campus had five different things to choose from in terms of where they wanted to focus. But it did shift much of the work from you to us, so that in the first thing, you did much of the development of the slides, which we all greatly appreciated. But there was collaboration, as you noted from the very beginning, beginning with a meeting in Oneonta before the start of the 2024 academic year. So, what were some of the other benefits or challenges that you observed from your side or from interactions with the groups?

Racheal: So, I think that, John, as you’re emphasizing, I love that we were able to offer more choice for engagement to our faculty. I think that was so vital. And the other thing that I would add to that, just being able to engage a group of faculty who were all coming not only from different disciplines but from different campuses and institutions. That was some of the feedback on our survey that faculty mentioned really enjoying. Oftentimes, we can get in our silos, we always talk about this. We’re in our own departments. We’re in our own disciplines. Here, I think faculty were exposed to the energy and excitement and fury and engagements and questions that people were bringing from so many different institutions across the state, and that was really cited as a strength, was just getting to have these conversations. So yes, getting to have these conversations with a group virtually from different locations, and also faculty often cited valuing just the opportunity to have these conversations and to do a deeper dive into materials that they could search out on their own, but they’re so busy, right? It’s like now they have the dedicated time to say, “Okay, I’m going to think about AI policies. You’ve given me a number of them to look at. I’m hearing what my colleagues are saying about the choices they’ve made, and we’re talking about what has worked and what hasn’t, and now I have this dedicated time to make those changes.” So that was another strength, I think. In terms of the challenges, and I think we could definitely talk about the Brightspace area and that logistical piece of how we ran this through the DLE, the digital learning environment, but I will just mention first, maybe before saying more about that, I do think that scheduling and logistics is such an important and also challenging part of a project of this kind. We tried to get ahead of that by scheduling everything in advance. So if we’re having monthly meetings with our communities of practice, we tried to set those schedules in the fall, or I think we had to circle back in the spring, of course, because teaching schedules change. But we scheduled out the entire semester at the beginning of each semester, and each of the five groups had to offer two meeting times in order to make sure that everybody could come to our meetings, and we did set high expectations too. I think that was important to us. We used backward planning, which we know from teaching and learning best practices. And so we tried to be really clear about the expectations up front, about attendance expectations and about deliverables expectations, because I think with our colleagues as well as with our students, we know that having those expectations set and those structures in place are what really allows you to thrive in common work together. So I think those things were really important, but they did mean that at the leadership team meetings, we were like, “Okay, I’ve almost scheduled my meetings,” so that was something we talked about.

Rebecca: One of the things that I think people would probably want to know is how did you share information across institutions? You talked about meetings, so that’s a synchronous opportunity. What about maybe some asynchronous ways that you were sharing information across institutions?

Racheal: We relied on Brightspace. Over the past few years SUNY signed on with Brightspace. We now have a shared multi-tenant structure that is used across campuses. There are a few campuses that aren’t included in that multi-tenant environment, and SUNY Stony Brook, although it does use Brightspace, is one of them. I mention that only because we’ll talk about both what Brightspace allowed us to do and some of the roadblocks that we ran into. One of which was that SUNY Stony Brook had to log into two different Brightspace areas, and they expressed that that was sometimes challenging for them. But I think that Brightspace was a good choice for us. And like with any project, each year you learn about things that went well and about things that you hope to do better on next time. And with Brightspace, our plan was to have a shared online location where all five of the virtual communities could go, both to access the materials we were providing in our communities, and also to engage virtually in discussion and to upload their final deliverables or assignments. So it was great to have Brightspace for that purpose, and it really did allow us to create some shared documents. Our surveys lived there. We provided lists of AI-related resources that multiple groups could use and did use, and we also wanted to make the five communities visible, so that if one person, say, was in the accessibility community, but they wanted to see what the Build a Bot community was doing, they could enter that Brightspace module. It was public to everyone in our area, and they could look around and check on what their colleagues were up to. To talk a little bit then about the challenges, there were two areas of confusion for us. Some faculty had trouble identifying the materials specific to their module, and they would sometimes go into the other modules and answer questions there, or say “where was that resource that you mentioned?” That’s one area that we would want to think about and address. And additionally, the submission tools that we used that first year there had just been one assignment dropbox everyone had uploaded. The second year with the five different groups, everyone was uploading student-facing materials, their deliverables to several different dropboxes, and then we were engaging in a peer review process with our leadership team, where we were just making sure that those deliverables hit the spot on the rubrics that we had been using throughout the professional development programming. And so there was some submission drama [LAUGHTER], but you know because we were all working together to review, give feedback to our faculty, check on our faculty, both within our groups and at our home institutions, we were absolutely able to sort things out and get people feedback where it needed to go. But I think in the future we would consider maybe using the groups feature in Brightspace so that each group would only see the assessments and materials specific to that group. And we also talked about how we’d therefore identify which resources needed to be available for everyone, rather than just saying like, “Okay, here’s everything you can explore it if you want.” Maybe just identifying, “Oh, here are some of the things I’m putting up, the bibliography or the links.” I think these should go in the shared area so that we could direct people through Brightspace in a way that was maybe more efficient for them.

John: The basic issue was that some people logged in, saw five modules, and assumed that they had to complete all five of them, and that led to some confusion. Now, I know in our group we did show people where they should put things and where everything was, but, like our students, sometimes people don’t always remember all those things because they’re doing lots of other things at the same time, and we did have some issues where submissions were going in the wrong location. But the flip side of that is it did make all the resources available to everyone, so in principle, each person could lurk and observe what was happening in all five of the groups. So there’s some tough decisions there. To what extent you want to limit the visibility to make it easier for people to focus on what they’re supposed to be focusing on versus giving them the option of browsing more broadly and groups may be a way of dealing with that with assignments, but I kind of like the way it was set up as different modules once people got that figured out. And I think after the first month or so, they pretty much knew where to go until the final submissions, and that’s where things broke down.

Rebecca: To me, it sounds like an added benefit here of faculty having the opportunity to participate as a student in the environments that they use for teaching.

Racheal: I think that’s a great point, and it makes me think too about the virtual aspect of these synchronous meetings. In our initial grant year, we had mostly run meetings of 10 faculty on our own campuses, and those were in person. We also had an element of connecting with cross-campus folks in a small group of three or four, and that was the element of the project that didn’t get as much positive feedback at the end. People felt like a lot of the focus was on the in-person groups, and it certainly was. So when we were designing the second year, and just thinking about the needs that we wanted to meet for our faculty, we decided that it made sense to move to that virtual model in order to make the choices available. And so, I think that we’ve had the added benefit, again, of really foregrounding that cross-campus communication in the positive ways we already touched on. But I do think you lose something with virtual sometimes, or it’s different than it can be in the classroom. So just because you’ve shown somebody something and demoed it during a virtual meeting does not mean that person wasn’t checking their email [LAUGHTER] or dealing with other work tasks at that time. I think we really all enjoyed the energy during the in-person meetings that our colleagues on our campuses brought. I think the virtual meetings also were very strong, and I felt like we built community within my group as well. But I do think with any virtual meeting, there’s just that built-in element of distraction and multitasking that we all are familiar with from the demands on our time.

John: One of the similarities between the two things is that there was a keynote speaker for each of those to help frame the work. Can you talk a little bit about the keynote speakers and what they focused on to help set the tone of the project?

Racheal: The first year we invited Anna Mills, who is a longtime writing instructor who works for an institution in California. Anna Mills often focuses on peer feedback through AI, and one of her primary goals and the reason that we asked her to join us as the lead speaker for our first project is that she approaches AI from a critical perspective, and she’s always trying to foreground learning objectives. This was something that ran throughout all of our planning. Many of us are instructors or professors, as well as working in educational development from a variety of different angles, and we know from being in the classroom, that it is all about learning objectives. That is where you start, and the decisions you’re making about AI are about how you either preserve or update those, now that students are often engaging with these tools. So Anna has a great set of critical approaches to that. And she also is really on top of changes in tools. That’s another request we often get from faculty. And John, you were just telling me about the great work you do on your campus to just demo some of the tools that students might be using as they change every semester, as agentic AI use increases. I don’t know if you want to say anything about that?

John: At the start of each semester, we do a workshop here talking about what things have changed since the start of the previous semester because a lot of people tried AI four years ago and kind of figured it wasn’t much of an issue because it did such a poor job in terms of hallucinations and pretty predictable writing patterns. But, the people who still believe that need to try it more recently to see what’s changed, and certainly agentic AI and browser plugins that will answer student questions automatically from a Chrome browser are things that faculty should be aware of, just because it is affecting work that many students are doing.

Racheal: Yes, and that’s one thing I admire about what you’re doing, and I admire it in Anna’s outreach as well. She has a big platform, and we were lucky to have her come and join us for that initial talk. In our second year, we hosted two talks through our grant funding. We hosted Marc Watkins, who is a co-author of the Norton Guide to AI-Aware Teaching, which is coming out this fall. It might have already launched. I know their ebook launched over the summer, and their hard copies will be out soon. He also did great work focusing on some of the updates. We heard from faculty that they had learned a lot from his presentation. He runs an AI institute for faculty at his home institution in Mississippi. So again, our goal with these speakers was to set a shared language and a shared critical perspective going into the work of the different groups, so we got a lot of positive feedback on Marc’s talk, and then we also did host Mel Hogan through funding associated with my group designing assignments for AI literacy, and Hogan’s talk focused on the critiques of AI that are often coming out of a humanities and social sciences perspective, and it was important to me to fund that talk because I simultaneously have been advocating for support for faculty who are having to adapt to AI, and I often hear from our faculty who are sometimes in disciplines that are more engaged with the political conversations, that they wish that critical perspective on AI was represented in some of the professional development opportunities that are available. Because again, we have faculty coming from so many different orientations, places, disciplinary formations, and so I wanted to make sure I honored that feedback. And so we heard from Mel Hogan, who is a media studies and film scholar. And Mel has been essentially sharing both at her home institution in Canada and more broadly in some talks she’s been doing a set of political and ideological concerns related to AI, and that was a second talk that we opened with.

Rebecca: You’ve talked a little bit about some of the opportunities for folks in the cohort to share and interact with one another. What are some of the things that participants generated that were maybe more widely available or more widely distributed as a result of the project?

Racheal: I worked with Ed Beck, my colleague here at SUNY Oneonta, who was also serving this year as the chair of the Faculty Advisory Council on Teaching and Technology for the SUNY System. Ed helped us build a WordPress site that acts as a repository for some of the student-facing materials and other deliverables that faculty made through our groups. So we definitely had faculty share within the smaller virtual communities, but then we also invited them to submit their materials to this site, which we included a Creative Commons license for the site, so the materials are there. They can be viewed publicly. They can be viewed across the system. We definitely did not mandate that our faculty share to this site. We know that intellectual property and teaching materials are personal, and people think of them in different ways; they use them in different ways, but we did invite folks to share. So there are examples from both cohort years on the SUNY ACE site, and we’re hoping to encourage additional faculty to post there as well, although we have some up now.

John: I know a survey was done at the end of this, and you referred to this a little bit before, but could you tell us a little bit more about the reactions of faculty participants to this project?

Racheal: One of the things that we wanted to track when we gave a survey before the program began, and then we gave a similar but slightly different survey at the end, we wanted to track how comfortable people felt engaging with AI in a teaching and learning setting, and the response there was really positive. People, regardless of their initial level, felt that they were more equipped, they had more knowledge, they had more time engaging with tools and ideas related to AI. So that was one aim that we set for ourselves, and the survey data did reflect that faculty felt more comfortable, more knowledgeable related to these themes at the end of the experience. So that was really great. The other celebration that came up again and again: people really liked having the time to talk to colleagues, having that devoted time to come together around a common theme, which again I think faculty experiences and all of our experiences at the university are so dependent on our departments and kind of who we see day to day. So having those opportunities to sit down, develop big ideas, hear from others how they are responding and what they are doing, faculty really appreciated the opportunity to do that. And as always, I like to encourage people in any professional development settings of this kind to steal ideas from each other. So, [LAUGHTER] we talked about that in my group, and I think that ethos of sharing all the materials and resources on Brightspace, of having conversations, of showing what you’ve done, of workshopping. We did workshopping in a lot of these groups, so we weren’t just asking you to design a new assessment or a new assignment. We were asking you to share that and have a conversation with other colleagues. That also was mentioned favorably, and we saw that cross-pollination of ideas and approaches, where some people could learn from and draw from others’ ideas. You might get one approach to a problem you’d been seeing in your class and say, “Oh, I love how this person was doing that. That’s what I’m going to do.” And sometimes these are just little tweaks, ways that you’re going to frame an assignment, things you’re going to say in class, how you’re going to revisit a policy over the course of the semester to really enliven it. And we saw that in the survey data that people appreciated that sharing opportunity.

Rebecca: Well, thanks so much for sharing this project and how it’s been developing over time. We always wrap up by asking, “What’s next?

Racheal: John and I were chatting a little bit about this before we started. I have not pursued IITG funding for another multi-campus initiative this year, so I’m focusing a lot of my energy on my home campus, and it’s been really rejuvenating to do that too. I loved working on these multi-campus projects, and I loved the leadership team members, meeting so many colleagues, presenting at CIT with colleagues, and sharing the great work that we’ve done. And now I’m also very excited to be here at home in Oneonta doing some different communities of practice, book groups, those are a lot of the things that I do here as the Pedagogy Support Specialist in the Faculty Center for Teaching, Learning, and Scholarship. This semester, one of the things I’m the most excited about, I’m working with the Center for Racial Justice and Inclusive Excellence, and my colleague there, Sally Hahn, who’s an anthro faculty member and also is leading the center for a term, but she’s leading a banned books fair for staff, faculty, and students, and we’re running a faculty reading group dedicated to Octavia Butler’s Parable of the Sower. We already have 20 signups. We have had a lot of enthusiasm around some of the novel groups that we’ve run, including a group on Nineteen Eighty-Four last academic year, and we’re asking faculty to read these books together to connect their themes and ideas to their teaching, their research, contemporary political and social questions, and we’ve gotten a lot of enthusiasm for that programming. So those are some of the things I’m excited to be focusing on now.

John: And you will also be presenting on these projects at the AAC&U. [LAUGHTER]

Racheal: Yes, John and I are traveling to Atlanta in October to present the work that our team has done at the AAC& U’s conference on AI, which looks to be an enormous higher ed conference. Lots of folks from many different institutions nationally coming, and we’re excited to present this work because SUNY is such a big system, and I think that our project models how to share resources in a way that creates more choices for our faculty, as we’ve discussed.

John: Well, thank you. It’s always great talking to you, and thank you for all the work you’ve done on these two projects.

Racheal: Thank you so much, John. It’s been wonderful working with you too, and just benefiting from your own expertise and the incredible professional development work you do on your campus.

Rebecca: Thanks so much for joining us.

Racheal: Thanks for having me.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing Assistance provided by Madison Lee.

[MUSIC]

452. Revisiting A Curated AI Framework

Shortly before the start of the spring semester, Tim Curry joined us for an episode discussing his ambitious plan to create structured learning experiences for students using chatbots. In this episode, we check back to see how this experiment developed. Tim is a lecturer in the Department of Health Sciences at Northern Arizona University. He is developing AI applications for autism research and tribal health service while building Northern Arizona University’s Applied Health Analytics program. He’s also working on the completion of his PhD degree in health equity technology.

Show Notes

Transcript

John: Shortly before the start of the spring semester, we had an episode discussing an ambitious plan to create structured learning experiences for students using chatbots. In this episode, we check back to see how this experiment developed.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Tim Curry. Tim is a lecturer in the Department of Health Sciences at Northern Arizona University. He is developing AI applications for autism research and tribal health service while building Northern Arizona University’s Applied Health Analytics program. He’s also working on the completion of his PhD degree in health equity technology. Welcome back, Tim.

Tim: Hi, it’s great to be back here.

Rebecca: Today’s teas are:… Tim. Are you drinking tea with us today?

Tim: I’m not right now, but I was, and the tea I had this morning was a spice blood orange tea. It was quite tasty.

Rebecca: Looked like it was in a beautiful mug.

Tim: Oh, yeah, yeah.

Rebecca: turquoise… ribbed…

Tim: Yeah, with a little bit of a, almost a goldish highlight on the ribbings.

John: A bit iridescent.

Tim: Yeah.

Rebecca: How about you, John?

John: I am drinking a Lady Grey today.

Rebecca: Oh, I have its companion, a British breakfast.

John: Very good. Earlier this year, Tim, right before the start of the spring semester, we discussed a plan that you had developed in which students would engage in a structured sequence of interactions with a chatbot tutor that you created in each of the course modules in your health science course. The course was scheduled to begin actually the afternoon of the day when we last talked. We’re following up today to see how this plan has worked out once it hit the actual classroom. First, though, could you provide our listeners with an overview of the course, because not everyone may have listened to that, or it’s been a little while, and just remind us of what the course is and how the AI activities were structured.

Tim: So, as a rewind here, so I actually use this in two of my classes. The first one being our HS 300, which is a class focused around the diseases we get as people and how we work in that space, in more of a public health focus. And then the second class was a fitness wellness class that’s focused on how do we do physical activity more of a clinical population focus, and it had that as the topic, and then it’s also our writing-intensive higher-level course. So, in both of the classes, we had a tutor bot. The way that it functioned was I gave them a file that I built out that was a learning file that the bot would read and would have an idea about what are the topics that the student needs to learn and then would work with them in a very Socratic style to help them learn the content and the depth that was required as kind of the minimum foundation points for them walking into class in the first day and they did actually fill in the questions I had in that file, and it was required to hand that in, as we could call it proof of learning.

Rebecca: So I’m picturing the first day. How did students react to your plan?

Tim: About what I expected, honestly, which was tentative. And that first day, and I’ll actually expand that, and say that first week, we had a lot of discussion in class about what does this look like, what’s the purpose of it, as we had, I believe, discussed about six months ago. One component of that first week was about roughly half of one of the first two days of class in that first week, they had built-in time in class for them to actually dive in, learn about this GEM that was the host of our bot, and actually get to start working with the first file for that first week, and get their hands on. And I’ll say that first day it was very interesting to hear about the reactions, and some students, within the first five or 10 were like, “this is not for me,” which is fine, because that was how it was built, was that they did have an option B that they could do without having to engage with the bot. Quite a lot of them had said, “I’ve never ever engaged with an AI like this before, and I’m a little uncertain, but I’m willing to go and give it a shot.”

John: Roughly what proportion of students chose to continue with it, and what proportion chose to opt for plan B?

Tim: Yeah, that’s a great one. And in fact, I’ll say that was one I was watching throughout the entire term, because they have to hand in this like proof of learning, which allowed me to kind of track what was the proportion. Roughly, the proportion was like 70% continued with AI, and 30% chose to go the more traditional text-heavy route. I’ll go and say this. I would argue in a 16-week term, I would say just off the top of my head that by about week six the students had settled into the way that they were going to engage.

Rebecca: That’s not too bad, about a third in.

Tim: Yeah, it’s not too bad.

Rebecca: How did it work?

Tim: Really well, I would say. Overall, there were definitely things, that within the first two weeks I hit the fun point that I’m sure all three of us have at some point, which is Gemini pushed a major change in their models and updated, and suddenly the AI tutor [LAUGHTER] wasn’t working the way I was expecting it to work. When I had trained it and done the last round of kind of making sure it was working, that was end of December, start of January. By the time we hit like week two of the class, the model had been adjusted and suddenly it just kind of wanted to go off in its own in right field and treat the student as if they were like a medical student and go in way too much depth. So after that, I realized very quickly in that week, too, like, “oh, we have a problem,” so very quickly went back, adjusted the system prompt, put up some more forms of constraints in a guardrail to rein it back in, bring it back in, and effectively the way I did it was to get the bot to do less explaining and focus more on explicitly what the student needed to learn and allow the student to drive where they wanted to engage further. And after that adjustment it seemed like that solved the issue and the students were able to better engage with the bot overall and I’d say overall, for both the AI bot and the textbook route, the big change of having these effectively notes that they were expected to hand in dramatically improved their readiness to engage in a flipped classroom, which is the way I teach, and so I would count that as a win period for the AI bot, the fact that at the end of the day the grades were effectively the same regardless of the way that they engaged. So I would go and put that up as a positive sign overall that the AI at least performed as good as the traditional way.

John: And in a first iteration that’s probably a very positive outcome, because often new experiments have some rough edges on them. I was especially curious about the performance of the Gemini Gem, because I was working with a group of 18 faculty in SUNY from various campuses who were experimenting with chatbots, and many of them started with ChatGPT, but then moved to Gemini, and they had very similar experience, that the GPTs in ChatGPT seemed to pay much more attention to the guardrails that were put in, while Gemini Gems seemed to go off in directions that were different from what the instructions were, and that caused a bit of frustration for several of the faculty, because one of the reasons they moved to Gemini Gems was because their campuses had Gemini, and they thought that would allow all students to have more equitable access, but they had begun in ChatGPT because the grant had funded Chat GPT accounts for them, but again students were complaining that the free accounts didn’t allow that much interaction with it, while those who had paid accounts had much better quality of interactions, so it was a trade off, and ultimately pretty much everyone who had that choice did stay with the Gems, but it did require quite a bit more training. So, I think your experience there was not unique.

Tim: And I’ll just go and say this as well. I did also one more adjustment I did in week three which further helped was, in Gems, you can have certain variations of it, right? Just go and turn on as the default that the student can go and turn off if they want, but generally they don’t, and I defaulted on the guided learning mode, and that further enhanced that Gem as well. 8:11

Rebecca: So it’s always interesting when you try something new and something goes awry, and obviously it sounds like you adapted well. How did the students adapt to this minor interruption, especially because it happened early on.

Tim: I think the biggest item in terms of the feedback was that I was very open that first week of “this is an experiment, we don’t know what’s about to happen. I don’t know, I know what I’ve done for my testing, no idea what’s about to occur here, and I want to know your feedback, so let me know. We’ll go and talk about this.” And opening up that window of this is an experiment, you are my group. If this doesn’t work, we’ll adjust and change. I believe that it allowed that space for a dialog to happen, and that first week we did half of one class where they had a chance to go and get in there, work on it. I got a lot of feedback from them in that class, then once we got in to week two, that feedback of like, “Hey, this took a really long time because it kept like wandering off over here, and I had to bring it back, and it was really annoying,” I think was the biggest response I had, and then after we got it tuned in, I began to hear more feedback about, “Okay, I kind of like this, because if I’m totally lost on what is this problem with the heart, why do I care about it?” It really allowed them to explore if there was an area or a disease that they really found fascinating, and it just gave them more freedom to go and explore that. So, overall, the response that I had from the students was, I’d say broadly, it was kind of two broad groups. One was: this was very intriguing, and it was about the same as what I’d expect if I read the book. I’d say out of the 70%, like roughly I’d say like maybe 10 to 20 of that was that group of like it was nice I didn’t have to pay for the book, and then the rest of the group was generally that they liked the ability to have that freedom to explore or to ask the bot for more help in a space as they felt need.

John: Going back just a little bit, you mentioned you have these chatbots that are trained as Socratic tutors on that module, and then students complete an assignment at the end. Was the assignment built into the chatbot or was it something that was a standalone, and how did you receive all that? Did you get a transcript of the interactions along with some responses or something else?

Tim: So what I did was I gave them a file that was in Canvas, and the chat bot understands how the file is laid out, and it has a set of questions and tables and cases and all of that stuff without the correct responses put in there, and so the student’s job was to start a chat with this bot and to give it the file, and so that the bot knew how to drive the engagement, and then what they did was, as they engaged with the bot, they would go through and fill out each of the components of this file, and then they would just go, and it was a hand in before the first day of class that week that we did through the Canvas site. Now, a big part of not only these classes, but just the classes I go and teach across the board is that I do a contract grade, so for these items, because the focus is on “I want you to engage with the content, I want you to learn it, so in class we can work on like going further and building out skills,” so they knew that as long as they completed all the work, it got marked as done. I wasn’t about to go through with a comb and dock them points for everything that wasn’t exactly right, but once they had that handed in, it unlocked the key for them, and then they were in charge of ensuring that they went through and reviewed the key to make sure that they were in the right space.

Rebecca: Seems like a good plan, when otherwise that would sound like a lot of grading.

Tim: It did work well.

Rebecca: And you said that the students who chose the textbook method essentially did the same assignment, right?

Tim: Yeah, so it was adapted for the textbook, but exact same idea. The plan, as well, was to keep the overall load on the students the same. It seems like, based on what they reported in evaluations, that was the case, that regardless of path, it was about the same amount of time to complete each one.

Rebecca: Besides the tech surprise, were there other surprises, things that you didn’t expect? Anytime you go into an experiment, things come up that you don’t expect.

Tim: I think the one that was unexpected and interesting, and I suspect both of you, being faculty, you’ll hopefully laugh at this, was HS 300, in particular, is a content-rich class. I mean, not only is it a 300 class, but human diseases, there’s a lot of them, [LAUGHTER] like there’s far more than we can fit in one class, so we only focused on functionally the major diseases for each of the body systems, and still there’s just a lot of content that you need to know, and the feedback that I had at the end of the day kind of made me laugh, which was regardless of the path that they picked, because now we were requiring them to show at least foundational knowledge prior to walking into class as a core component of this class was that they were spending three to four hours a week outside of class doing all this work and going and like preparing for class and that it was more load than what they were used to or expecting and a lot of the feedback was “and I feel like I learned so much more because of this.”

John: That’s a nice positive outcome. We don’t always get that. We often get the “this class is so much more work than I expected,” but it’s not always combined with “and it was worth it.” That’s a really positive outcome.

Tim: Yeah.

John: You had mentioned that you were using a flipped classroom approach, and earlier when we talked to you, you mentioned that you were using active learning activities. Could you talk a little bit about what a typical class would look like in terms of a typical day in the classroom?

Tim: I’ll actually expand, I’d say a typical week, because usually things span like day one to day two. So, a typical week, the first day of class, we go and start off, and the first thing that we do… well, after the “hey, this week in class… , and don’t forget about this,” all of that fun at the start of class was we would do a tRAT, and for those who don’t know, a tRAT is a team readiness assessment test, and it is classically done with these cards that look like a card I’d buy for a lottery game, where we would code effectively a multiple choice quiz to the card, and then they work as a team to go and figure out what do they think is right. If they’re not right, they have to keep working on it until they go and get it right. I’ve created a digital version of that, so we begin with that, not only to kind of like warm their brains up, because I would get them Monday evening at 5:30.So, a lot has happened. [LAUGHTER] So that helped them kind of get their brains back into our content for the week and help them understand, like, “What do I recall now? What do I need to go work on?” and then we would dive into one of two things, if we were having a bigger application week, which was a lot of the more impactful work, we’d walk through, here’s our work for day one, day two, here’s our task for the end of the day, and then they would just dive right into it, and in those cases, day one was like they were working with a small, and by that I mean no more than four of them in a team, a small team of their peers, and they would work on doing more of the engagement part, and then day two would be wrapping up that, and then doing usually a presentation to go and share out. So, an example of a week, this one was a hit. It was new in this format, and it seemed like it went over well. When we were talking about diseases of the lung, the pulmonary system has a lot of diseases that are infectious, a lot of ones that we would have to report to the state if you happen to get it, and even if it isn’t directly involving the lung, that might be the way it’s spread. So we wanted to introduce them to the fact that there are these things called an investigator that works for the state, and all they do is investigate potential forms of these that might spread that we might need to do some public health work on. So in the first day, they were handed their initial case that they had to go and talk with this AI bot to figure out what had the patient done, where were they at? They knew that they had TB, but they didn’t know anything else. And then they were allowed to expand their investigation to potentially up to three additional bots, and they were told you might have to talk to all three of them, you might only have to talk to one of them. You have to figure this out on your own. You have to do the investigation. So they went through the path and the process to go through and investigate. As I’ll just go and say they did have to talk to all three. The first one went and led them to two of the three bots, and then one of those two went and led them into the last bot. And then they had to go and figure out what do we think happened, and then our day two of class, they wrapped up that work, and we had a chat as a whole class about what did you find out, how do you think this spread, what do we need to do next, and it was actually quite a complicated case, but that’s the way that it spread throughout that week was to go and show them, like, hey, this is an airborne thing, it’s a dangerous thing, and this is a real job you might be able to actually get, and by being able to engage with the AIs as effectively a patient, they had to go through the process of the fact that one of the bots was extremely not helpful, in fact, actually almost like was very rude, and the reason for it was because of the circumstances built into this bot if they were found out to have a TB test that was positive, they had significant potential losses that would be a major problem for them. So that allowed them to actually engage and attempt to work through real-life experiences that they might actually have to go and deal with.

Rebecca: That sounds like a fun activity, that of course leads me to want to know how did you build those bots? … and a little bit of detail about using AI in that context.

Tim: Yeah, so using AI in this format is, I’ll say for myself, has not been a new thing, it’s actually been something I’ve been doing for a while. It started for me out pre- 2022 and the world exploding and changing on us, but it started before that, because in my exercise physiology course I used to teach, engaging with a client is really important, and learning how to do that, so back in the pre-AI time [LAUGHTER] I had used IBM Watson bots that you had to build very crudely when we go and look at then and now, where they had to work with this fake form of a client to pull out all the risk factors and try to understand what were their goals and how do I work with that, and it worked okay. It didn’t work great. It worked okay, because of just the inherent limitations. After we got into our current era, the last four years, roughly, that expanded greatly, and what I would do is I’d build out a prompt for the GPTs first, and now for our gems that included all of the different things I wanted, as like building, so it did include things like here is the background of you as a person. Here is a personality. Here are the, for example, in this case, if they had had any type of testing done, what are the tests? What are the results? What are the tests that we don’t go and know about within all of those? What are the ones that you immediately share? And then there might be additional forms of like a test, like maybe an X-ray of the chest, that unless the student asks, the bot won’t just offer it up. So, what are you allowed to just go and bring up? What are you allowed to bring up only after they go and ask about this thing? And then I did find that when we get into the disease space, where I want them to model the signs and the symptoms of a certain form of a problem. I need to give them exactly what are the symptoms you’re feeling, and I never give them strictly the textbook, because that’s usually not the reality. They’ll have some, but not all. And then I found that it was also very important that I put a very strict box around the bot, this is what you have, so you are a person who has a problem A, it occurred because of B, C, D, E, F, G. You’re not allowed to share any of this. It’s only for you to use to better drive your responses to the students, and so that as they ask about a thing like, have you had a fever, if I hadn’t defined that, it at least knows which way it should kind of push.

John: One of the things you had mentioned when we talked last time is that you had students engage in some self-reflection, and you referred to this a little bit earlier. What did they reflect on? Was it a metacognitive type of reflection, or was it reflecting just on the material? Could you tell us a little bit more about that?

Tim: So this actually, I won’t say that I invented it, because it’s not mine. I actually borrowed a resource from the American College of Sports Medicine annual conference three years ago in Boston, where there’s a really good discussion about flipped classroom approaches within the exercise physiology space, and one of the items that was brought up that I borrowed because it had been used in some research work was this weekly graded survey. So, in the Canvas shell, it’s arranged as a quiz, but it’s marked as a survey that is graded, and all that that means is once they go and do it, it’s just marked as check done, and it runs through a range of different things, and it’s the very last thing that they’re in charge of doing every single week of the term. So it begins with exploring how was the week overall, and I don’t say just for this class, I just say how was the week overall for everything, and that’s a multiple choice, and then we go and walk through, how well did you feel prepared for this week in class? How was the week for you overall in your life? Because we all know that we teach folks who are increasingly more stressed out right now, like I don’t know about both of you, I might have brought this up before, but I have students who, 50% plus of my class, is in six to seven classes and working one or two part-time jobs. They are incredibly busy, and then there’s other life that happens on top of that. So, having a feeling of like maybe they were off this week and maybe their life just took a bad turn in this week, so that makes sense. And then we went and moved into some more open-ended stuff, which included: What did you learn this week? What do you have in terms of a question? Were there any types of like a moment that was an aha for you (it’s fine if there weren’t)? And then we do a bit of goal setting at the very end, which is: “What do you want to do next week to improve on what happened this week? And then when you look back at this week: Did you accomplish the goal that was set for the current week? And then finally, just kind of a broad: “Anything else you want me to know, like did your car break down on the drive up from Phoenix, and you were stuck on the side of the highway for a whole day? And it’s been a thing that I’ve used for a couple semesters now, and overall it seems like it works very well, and the feedback has been very positive, in that a lot of the students are like, “I’ve never had a class require me to, and I’ve never thought about, stopping at the end of the week and reflecting over how things went,” and it’s helped them with a thing like, especially my students, who are those “got to have an A” students, which was me, in fairness of, they might have a bad week, and then when they have to stop and reflect, they’re like, “Oh yeah, I had to randomly work more hours than I thought, and I found out that my dad has to go in for a cardiac stent, of course it was a hard week.” And so it helps them kind of put the week in context better.

Rebecca: In the spirit of reflection, now that you’ve had [LAUGHTER] this experience over a semester, would you continue the approach? What are some things that you might do differently next time? This is your moment to reflect. [LAUGHTER]

Tim: Yeah, so reflection for me would be this, I think I wouldn’t, and I’m not planning on moving forward in exactly the way that we did. Learning that just the fact of Google will push adjustments to their models regularly, and it’s not always easy to go and track it down, that makes it a little bit unpredictable, and I think there definitely is time that could be clawed back because there are times when AI was just inherently not efficient. So in the current model I think there are improvements that could be made, and how I would improve this is I am very much a fan of OER, especially at the current time when things are just expensive, so I would want to create and provide content that provides that foundation that is a little bit more controllable, so I have a better idea of like this is the time that they’ll spend here, so I don’t say like it’s going to be an hour or two, and then bringing in that AI in a more limited way, but in a higher impact way, where instead of using the AI to learn, they’re going and using it as a way to apply and test their own knowledge in a manner that’s going to give them immediate feedback and allow a challenge to the way that they actually approach the content for that week.

John: One of the big concerns with the use of AI models is personally identifiable information. Were you concerned about that issue? I know the contract that Google provides with campuses is that they don’t use the data for training, but were there any residual concerns about that issue?

Tim: This is a great point, because this is a big thing that I work with our faculty on a lot, because with AI’s advancement so fast, it’s so easy to ignore the legal terms of use, which I know everybody reads in fine print. [LAUGHTER] And so for this spring semester I was not worried about that, number one, because as you’ve alluded to, we have a Google ed license, which explicitly prohibits them from training on our chats that are through our accounts, and as a part of that, I made sure that we had a pairing up with that, with ensuring to be very clear, I want you to log in through your college accounts onto this, that’s the only way that it’s like a guaranteed private, and part two, even with that, I don’t want you to put in anything that you wouldn’t mind having as like a TikTok post, as like a public thing. It’s here to help you go and learn the content. It’s not here to help you troubleshoot why you’re not [LAUGHTER] feeling great today. So I did go and build in a bit just to help our students better understand, like, what are the safe and unsafe ways to engage with AI in this format, because I don’t think that’s talked about enough in the world. Now, with that said, I will say that there’s a very interesting change on the horizon, and that’s because our institution has been… for quite a number of years now. We have both Google and Microsoft. Don’t ask me why, I don’t know why [LAUGHTER]… but moving into this fall we’re moving to purely a Microsoft license, so we no longer have the Google license, which has been an interesting change to find out about, because it’s been very recent that we’ve found out about it, that really affects this work. So, moving forward, we, we being myself and the faculty who go and teach these classes, we’re going to have a conversation over the summer about how do we manage this, because the Microsoft format is through the Copilot, and unfortunately the way that it works is I can make these – they’re called an agent in place of a Gem, however, once I go and push them for a student, at least right now, from what we’ve been told, it costs us tokens, which means that there’s an ongoing cost that’s not like a Gem, where it doesn’t matter how much they engage, it’s all wrapped up in that fee. This is a big change, because, as you know, as I suspect, both of you are feeling like finances are a worry for a lot of institutions, so we want to be very thoughtful. So we’re having to go and talk about this, about how do we move forward in the future, is this a thing where we keep it as a G em, and we’re just explicitly clear about you should put no private information in here, because even with a free account, there’s a fair amount of use you can do with these, so that’s a big change we have to go and figure out as we’ve moved away from the Google group.

John: That’s really unfortunate, because I know in campuses where people have had both the choice of faculty is to use Google tools and to use Gemini and Gems, both because of the cost and because Copilots development is behind the others. Even though it’s based on the Open AI model, the training and everything else is unique to Microsoft, and it’s designed for a business environment to keep information locked down and also to build based on the use of the chatbots created by the businesses that are running Microsoft, because it’s designed more for commercial enterprises than for academic enterprises.

Tim: Yeah, well, and I’ll just say, as an expansion, too, because depending on the institution, I mean, one of the things I want to look into as well is there are increasingly more open source models that we can use, and there’s plenty of techniques that, if you have the compute capacity, you could potentially train a larger model, just going and thinking, like maybe you train a Llama 4 and use that as a teacher to train a student that’s smaller that then we can host locally. So there’s plenty of alternates, but it requires you to have the proper not only compute and server capacities, but also the fact that you understand how to make that work. So that is a thing I want to look into, because I do agree with you, Copilot is much more on the business versus straight up education side. So it’s going to be interesting to see how this changes, and perhaps that’s a point to go and have a chat about in like a year after we’ve had a year to explore how have we adapted to the Microsoft model.

John: And that is probably going to be a much more of a transition than adjusting to the change in Gemini.

Tim: Yes, it will be. [LAUGHTER]

Rebecca: It’s a good reminder how quickly this landscape is shifting in the role that the changes in technology have, the role that these companies play in what we’re doing, the finances around it all, they all have an impact on what we do in the classroom. We always wrap up by asking: “What’s next?”

Tim: Yeah, [LAUGHTER]I think I might have alluded there. So, what’s next is for this fall, for both of the classes where we’ve used this in. First of all, one have to figure out what to do with the fact that we will no longer have G ems in an institutional license, so have to figure that out. And then two, how are we going to do adjustments with the classes to provide content in more than just text, because that gets a little bit boring. So, how do we provide the content foundation, and then back it up with applied learning through the use of an AI to further enhance learning? So, we want to keep moving forward with this idea. We just have some hurdles we have to go and jump over first.

John: It’s going to be a challenging experience moving forward, but I hope things go well.

Tim: Me too. [LAUGHTER]

Rebecca: I hope you’re a hurdle jumper, because they’re going to keep coming. [LAUGHTER]

Tim: Yeah, I think I might be sore by the end of this fall. [LAUGHTER] It’s going to be interesting to see what happens.

John: Going back to the multimodal thing, one of the people I worked with this year from one of the other institutions in the health sciences field had put a microphone in a mannequin and a speaker there, and they had voice interactions with a patient who was basically just a plain mannequin, but they were interviewing it and asking questions. They enjoyed the multimodal interactions, probably a bit more than just text interactions.

Tim: That’s a really interesting idea. I might have to follow up with you more about that, because that’s a really clever idea.

John: It wasn’t mine.

Rebecca: I immediately saw your gears turning. [LAUGHTER]

Tim: Yeah.

Rebecca: Well, it’s always a pleasure to talk to you, Tim. Thanks for joining us again.

Tim: Yeah, thank you. I’m sure we’ll follow up again in the future about how did the transition go. [LAUGHTER]

John: You’re not alone, because institutions are cutting costs, and many institutions, including us, have access both to Microsoft and to the Google tools, so we’ll see.

Tim: Yeah, we have some strong feelings as faculty.

Rebecca: Yes, we do. [LAUGHTER] All the big feelings. [LAUGHTER]

Tim: Yes.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

451. All Day TA

 Imagine having a virtual clone of yourself to assist your students whenever they need help and would track where students are struggling. In this episode, Joshua Gans joins us to discuss an attempt at creating an AI platform that accomplishes this.

Joshua is a Professor of Strategic Management and Jeffrey S. Skill Chair of Technical Innovation and Entrepreneurship at the University of Toronto’s Rotman School of Management with a joint appointment in the Department of Economics. His work focuses on the economics of innovation, entrepreneurship, and artificial intelligence. He has authored numerous scholarly articles on the economic impact of generative AI and, with Ajay Agrawal and Avi Goldfarb, of Prediction Machines and Power and Prediction, two books that have helped shape public understanding of the economic implications of AI.  Joshua has been thinking deeply about what generative AI means for higher education: how it can support teaching and learning, how colleges should prepare students for AI-augmented careers, and how AI may reshape labor market opportunities for today’s students. At the University of Toronto, he and Kevin Bryan developed an AI teaching assistant, All Day TA, which has been used to provide students with support outside traditional class hours.

Show Notes

  • Agrawal, A., Gans, J., & Goldfarb, A. (2022). Prediction machines, updated and expanded: The simple economics of artificial intelligence. Harvard Business Press.
  • Agrawal, A., Gans, J., & Goldfarb, A. (2022). Power and prediction: The disruptive economics of artificial intelligence. Harvard Business Press.
  • All Day TA

Transcript

John: Imagine having a virtual clone of yourself to assist your students whenever they need help and would track where students are struggling. In this episode, we explore an attempt at creating an AI platform that accomplishes this.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is

Joshua: Gans, Professor of Strategic Management and Jeffrey S. Skill Chair of Technical Innovation and Entrepreneurship at the University of Toronto’s Rotman School of Management with a joint appointment in the Department of Economics. His work focuses on the economics of innovation, entrepreneurship, and artificial intelligence. He has authored numerous scholarly articles on the economic impact of generative AI and, with Ajay Agrawal and Avi Goldfarb, of Prediction Machines and Power and Prediction, two books that have helped shape public understanding of the economic implications of AI.

Joshua: has been thinking deeply about what generative AI means for higher education: how it can support teaching and learning, how colleges should prepare students for AI-augmented careers, and how AI may reshape labor market opportunities for today’s students. At the University of Toronto, he and Kevin Bryan developed an AI teaching assistant, All Day TA, which has been used to provide students with support outside traditional class hours. Welcome,

Joshua:.

Joshua:: I’m happy to be here.

John: We’re very glad you are joining us. Today’s teas are:…

Joshua:, are you drinking tea by any chance?

Joshua:: I am not. I’m sorry. I see it in the title. I don’t see it in the instructions for the podcast, and I would have followed them. [LAUGHTER]

John: Most of our recent guests have not.

Joshua:: I guess the listeners out there should pause this now and go get their tea.

Rebecca: Sounds like a good plan. I have English afternoon today, John.

John: and I have an Irish breakfast tea today.

Rebecca: Going all imperial. [LAUGHTER] At the initial release of ChatGPT in late November 2022 a primary concern of faculty was really on the impact of generative AI platforms on academic integrity, but now that we’ve had more experience with generative AI tools, what have faculty and institutions learned about the risks and opportunities of generative AI?

Joshua:: Well, those initial concerns that people were worried about have only got worse [LAUGHTER] in many respects. Nothing in the development of AI has brought any comfort to those concerns, at least insofar as the foundation models are concerned. I think this was always on the cards, which is one of the reasons we started working on this problem back in November 2022 when ChatGPT came out. Put simply, it’s not just a student issue. ChatGPT helps you do stuff, and it helps you do stuff that you don’t like doing. It just so happens that we have set ourselves up for a challenge there with regard to students, because we give them a whole lot of things to do, for most of students, most of the time they don’t like doing it, [LAUGHTER] relative to other things they could do. And I think they very much understand why they should be doing it, but unfortunately, our students, it turns out, are quite human, and so a temptation is right there that they will take it. That said, there are actually things we’ve learned from this experience. I think actually students have been learning faster than faculty about AI. One of the challenges we have is, well, how do we prepare the students for the world that they’re going to be working in, where they’ll have AI tools, etc. when we, as faculty, and everybody else doesn’t know anything about that, [LAUGHTER] and really it’s very difficult to even anticipate. Well, one of the fortunate answers is that, amongst all groups in society, students by this very mechanism have actually spent more time with AI than most people, [LAUGHTER] and so they are more keenly aware of where it works and where it does not work. That may still pose a challenge for assessment, but I think students I’ve noticed have got better at using AI, even in their assessment, which has got to bode well for something.

John: We give a lot of workshops here, and there’s two sort of responses we tend to get from faculty. Faculty tend to be really excited about all the things that AI can do for them when they learn about them, but they’re really troubled about all the things that students can use AI for. What sort of changes should we be making in teaching practices to deal with an environment where student access to AI tools is pretty much ubiquitous?

Joshua:: I think, when it comes down to it, the problem we have is the problem we’ve always had, [LAUGHTER] which is how do we engage students and get them to learn for their own good. Now our tried and true method is to basically hold a set of carrots and sticks, or some mix between the two, and then come up with surrogates for learning, which is essentially not proof that you’ve actually learned or deeply understood, the things that are correlated with it, and they may be more correlated with it, or more appropriate for certain types of students, and certain types of learning patterns, and neurodiversity, we call it, and things like that, but that’s always been the case, and so basically our approach has been: you should learn this or else, [LAUGHTER] and the “or else” is also done with a caveat as well. It basically says “I’m a professor, I can come and I can give you lectures and talk with you. I can spend a little bit of time in office hours if other people have additional questions, but by golly, I hate grading. I hate setting assessment. I hate grading assessment. So I’m going to give you assessment a couple of times a semester, and we’re going to be all in on that, you either do well there or you get whacked with a stick.” Now, that creates a very high-stakes, high-pressure environment, but I don’t believe there is any scientific merit that that’s a good way to learn. The way to learn is the way we actually assume students learn, but none of them do, which is every week you prepare for the lecture, you do the readings, you come in and you pay 100% rapt attention in it, and then you repeat. That would be the nice way to do things, [LAUGHTER] but we haven’t matched the assessment with that at all, we’ve made the assessment to do with cramming for final exams and things like that. Now, to be clear, if you do do what the professors want you to do and pay attention in a methodical way, you do quite well. And you know which group of people actually did that? The professors themselves, when they were studying, they generally tended to do that. That’s how we select professors. So we basically created a mythical system with practical implications that are very predictable, and I don’t like to condone cheating, but if you tell people their livelihood depends on something, which is a critical moment, and they haven’t learned all this stuff, they’re going to do things. [LAUGHTER] I’m an economist, and that’s what we’d expect, and we’ve set up everything about assessment, actually, on the presumption that they will try and do those things as well. So, throw in ChatGPT, [LAUGHTER] you’ll get more of that. The sort of good news is that ChatGPT, and AI in general, is so powerful. We’ve got a whole group of professors saying, well, I guess I’m going to have to spend more time creating exams and proctoring it, and some people are talking about oral exams, and a whole lot of things. I guess I’m going to have to do all that, but the alternative is, or maybe not. Is there a way we can actually use the AI in a way that will actually alleviate the fundamental problem, which is high-stakes assessment, and so that was our motivation for creating All Day TA. We haven’t got to that aspiration, but that’s where we were starting from. Is there a way that we can reduce the incentives of students to go to ChatGPT and instead pay attention to the things that the professor wants them to pay attention to.

Rebecca: What I’m hearing you describe are faculty that sound a lot like students and students that sound a lot like faculty.

Joshua:: They have a lot of commonalities. [LAUGHTER] If you are honest with yourselves, we are all balancing all sorts of things, and in many respects, depending on your perspective, they can look the same way, which is one of the reasons why I don’t jump up and down with the moral force and outrage of some [LAUGHTER] of my colleagues, because we’re not exactly pure in this endeavor. So you can think about it, what if we were so pure and given that as professors we actually wanted to do the right thing by learning, what would we do? We’ll spend all our time on it. Then we might come up with something closer to what they do in Oxford and Cambridge, which is they have two or three students a classroom sessions. They teach all the time. They are able to monitor the students’ progress every week. It’s in an intimate setting, so everybody’s got an incentive to study and learn as they go along, and then who cares about the final exams? [LAUGHTER] They probably don’t mean anything whatsoever, and who cares about AI, because you’re sitting right there, you can talk to people, you can find out if they know things. So we know that if we alleviate the chief cost for faculty, that is their own time, then we can design the right system. So then there comes the question is: “Well, can we use AI to do that same thing?” And our answer is yes. [LAUGHTER] Yes, I think you can, because it is possible to take your own material, the things in the way you like to teach, transcription of your own lectures, add other stuff to it, and I know it’s not a popular phrase, but you sort of create a digital twin of yourself, and it’s a twin that can interact, and 95% of the time will give exactly the same answer to the students that you would if you were spending time there. But there’s a big difference, even in this ideal Oxford Cambridge world, I only have 24 hours a day, seven days a week, even in that ideal world. When we first did a very crude version of our chatbot for our own courses, trained on our own material, back in 2023, we had 200 students in the class, and they asked, over the course of semester, this chatbot, which is ChatGPT 3.5 which is like the very first version, 10,000 questions. There is no professor at Oxford and Cambridge who is answering 10,000 questions, and so you can see that. And now most of those questions came near the midterm and near the final… what a surprise… because it’s actually a very useful tool, given the current system, but at the same time, when we’ve done things since then, you can now use these same materials to generate questions to ask the students, and they use it for self-testing, and it turns out they like that even more, and that’s not even assessment, that’s just practicing for midterms and finals, they like that even more. And we got up in one class in Texas was 80,000 of these, they were using it all the time, and they’re very high quality because they’re trained on your data and with your learning objectives in mind, so it’s only a small step away from that to saying, “Why don’t the students do something a bit more like the Khan Academy does for elementary, middle, and high school, where it does little tests, and you don’t get to learn the next bit or do the hardest stuff until you’ve mastered the other thing.” What if we had these multiple choice short answers, or whatever things we might evolve to, if we put our minds to it every week, and then the students will be sitting there: “Well, I could go to ChatGPT for these questions, but the problem is by the end of the semester, ChatGPT won’t be able to answer them, and so I’ll need to answer them. So it may be just easier for me to just answer them now when they’re easy [LAUGHTER] and spread out my learning of it.” I don’t know if that’s possible. We have another thing we deal with in this, which is IT departments and integrations in universities. We haven’t had that sort of access to really put in the tests, but that’s the goal. The goal is, if we could alleviate the professor constraint, we can have low- stakes assessment at regular intervals, paced at the student’s own pace, that allows us to do things, for instance, like making it not by pass and fail, keep doing it until you know it, which is very low-stakes indeed. So that’s why I think there’s a lot of promise, but we’ve had a wake-up call. But the wake-up call reaction is not to run away from AI. I think it’s to run directly towards it.

John: Now, does All Day TA track student progress and provide reports to the instructor, or is it just something used by the students themselves without that sort of information. You’re getting some information on the number of uses, but do you have other information about whether people have attained mastery?

Joshua:: Oh, yes, we definitely do have other. So we don’t identify students by their questions and their performance, and you could imagine a world where some, for actual assessment, they give permission, and of course those get recorded, but there’s a reason for that. Is because we saw the sorts of questions students were asking, even if we didn’t know who they were, and like 40% of them were ones that they would be embarrassed to ask a professor, but it’s so much easier to ask the chatbot than to ferret around in their notes and textbook for the answer. So it’s really a smoothing tool, but if they knew the professor was looking over them, they probably wouldn’t want to do that. You want to encourage the students not to fear this thing and not to worry about, but at the same time, look at all this information you’re getting as a professor about what the students are asking about, where they’re struggling, and there could be also things that you expressed incorrectly. If the chatbots answering something incorrectly, chances are we know it’s because you failed to describe it properly, [LAUGHTER] and not because of the chatbot making stuff up. That’s not the way we’ve designed it. So, we provide basically, like you would with a really good TA, you’d sort of say, “Oh, where are the students struggling?” So, every week you get a summary of what they’ve been looking for, what they’ve been concentrating on, or where the difficulties are. You get access to all the questions, even though you don’t know who’s asked them, and you can look through all that. You can adjust your lectures next week because of the information you receive. You can also adjust the answers right on the spot, and say, “Oh, that’s not the answer I want to give here,” and I can adjust that. It all depends on how much time, of course, you want to spend with it, but all that information is there, and our experience is that professors love getting real-time feedback on what students understand. It turns out, for all our wanting to save time on things, we are interested in using our time well.

Rebecca: You’ve indicated a little bit about how many questions, the sheer quantity [LAUGHTER] of questions that students have asked. Do you have more feedback on how students are responding to the platform, and maybe what profiles of students are benefiting the most from All Day TA?

Joshua:: So, it’s varied on things. We’ve rolled out this platform that can be used in any course, regardless of the discipline, regardless of the subject, where it really has a nice fit, especially. And basically, the thing that gives them a nice fit is where those assessments are very high stakes, ironically, where you’ve got midterms, finals, and increasingly so at the moment, or where you’re in a profession like nursing, where not only do you have to do an exam, you’ve got to score high on it, very high usage of All Day TA for those circumstances. Where people have a PhD course, where they’ve got the assignments getting done, and other courses, which are a bit more discussion-oriented, and things like that, not so. It all depends. Now, we haven’t done nearly enough to know all those gradations, but we do know that as people currently move back towards in-person exams because of the AI issues, at least until we sort out something else. All Day TA is a huge component for that, because it allows you to do that without getting simultaneously overwhelmed with students in those exam periods, panics, and what have you. And we haven’t had a study of this, but I imagine also it alleviates student anxiety. I mean, we notice that most of these questions are coming out outside of work time, and that’s why we called it All Day TA, [LAUGHTER] because it really is around, and it’s there when the students actually need it, and so I think it can reduce anxiety just by its mere existence. Even though I would like it all to evolve into another system.

John: You mentioned this evolving into another system. Where would you like to see this go in the future?

Joshua:: Well, our business [LAUGHTER] is convincing universities to adopt this. I don’t know what your impression of university is, but fast to adopt technology is not their characteristic. The students are quick, [LAUGHTER] but the universities tend to be slow. Imagine you could integrate this into the LMS, and the students, you had all their courses and stuff like that. Imagine they’re taking a third-year course and they’re studying for it, and the previous prerequisite was in first year. Can you imagine that they have access to their first-year TA still in third year? That’s got to be a good thing. And then when you start thinking in those terms, you say, “Hey, why would we stop when the student leaves the university?” So the possibilities are quite enormous in terms of being able to provide something better for the students to learn and to work, taking into the very real account that they’re human beings [LAUGHTER] and have those sorts of issues with them. And so the immediate thing that I want to do is, I would like to, and we’ve had a couple of successes in this, being able to integrate right into the LMS at an institutional level. Fortunately, AI is actually not that expensive in terms of it requires authority, because doing things with LMS should require authority, but it’s not like a big budget item. [LAUGHTER] It’s actually quite small for what it is doing, and so I would like to do that, because I think that’s the path to providing a better student experience, but also providing the information we need to design whole curricula and other things. What is being remembered? Wouldn’t it be great to know in your third-year class what people are asking about on day one about the first-year class? Wouldn’t that be really helpful to know what they have retained and what they haven’t, and just be aware of it? We currently do all this kind of blind, and then once we’ve done that, working out how to put the assessment in, and the low stakes-assessments, and other things, is all possible, but at the moment, access to student records, and student assistance, and student privacy is extremely important, and we think it is the hallmark of this business, and these sorts of products, so it needs universities to be very proactive in wanting to do that.

Rebecca: One of the things that you just hinted at a bit is the idea of access to AI beyond the classroom and in jobs and beyond. So many college students are constantly expressing concern about the impact of AI on their future jobs, and what occupations might AI tools serve to complement, and where might they serve as a substitute for highly educated workers?

Joshua:: Oh, well, that is a very big and important question that we haven’t got a resolution to. We could have an entire other podcast about that issue, but the way I sort of thought about it in our modest example was, you’re a student, you’ve started your new job, you’ve had some on-the-job training and other things like that. Then something comes up, and they’re assuming you remember whatever, and you don’t. [LAUGHTER] So you can go to the internet, but it may be something that you recall your professor having talked about, and so you could then basically ask your professor again without emailing them for those things, and it could be brought back up, and it triggers the memories, and it makes those connections better, and it might just be much easier to take the knowledge that we’ve given the student, pour it into their head, force them to cram, but maybe they’ll just remember a bit of it, and they can pull that straight into the workplace and make them more productive on day one. That’s the kind of thing that I just feel we seem to say, “Well, you’re on your own,” [LAUGHTER] and everybody’s talked about lifelong learning and other ways of staying connected. Well, here’s another potential opportunity here that I can see how it might work.

Rebecca: Or an opportunity for quality control over the sources,

Joshua:: Yeah. Well, what if there are updates as well? It could be that, yeah, you learned it this way, but now there’s some additional things. I don’t know, the possibilities are endless. And, of course, this is not something any centralized company is going to be able to provide. It’s going to be the people who taught you in the past who are updating their own systems, and so long as you can push that out. It’s interesting, though. I think there’s a lot of faculty members who are concerned about such things. [LAUGHTER] I can imagine dire scenarios in which that occurs, but I have to admit I still have more than enough faith in what we add, in both developing content and also structuring it in nuance and thinking about learning and stuff that the replacement option isn’t there. I’m all about replacing the stuff that we don’t like doing, so we can concentrate on the stuff that we do like doing.

John: And colleges in general haven’t always done a very good job of encouraging that lifelong learning, because, for example, they set up systems to buy back student books, so that students don’t get to keep their books, and LMS systems, almost every institution will lock them down shortly after the semester ends, so all the material that they had and resources disappear. The practical reason for that is, if students have to retake a course, they don’t want them to have access to similar assessments, but for most students, having access to those resources would be really helpful, especially if that could continue after they left college, as you mentioned.

Joshua:: I mean, again, think about what having access to recent assessment is. Why is that the case? That’s the case because we kind of like to reuse things that work, but with AI, you can tweak it up enough that that wouldn’t be an issue either. And also that presumes that these are high-stakes assessments that people want access to, as opposed to low- stakes ones that they don’t. So, really, from my perspective, you can really see how we could push to a new system given the right set of circumstances. I’m hopeful one day one university will sort of like realize this and we’ll be able to evolve to it and really experiment with it. We’re doing a lot of that here at the Rotman School, but even here, where it’s been in our classrooms for three years, and a larger number of classrooms, we’re still not integrated with the LMS, because that’s controlled by the University of Toronto, which is one of the largest universities in the world, [LAUGHTER] and I must admit, with the current security issues that hit Canvas, no one’s jumping up and down to do things, so it’s a challenging environment.

Rebecca: You spoke a bit earlier about the concerns of faculty not necessarily having a lot of experience with AI, but still needing to help students prepare for a workforce that does use AI. Can you talk a little bit about additional ways that we can better prepare students for that work environment?

Joshua:: That is a very difficult question. One of the problems is that things are changing so quickly, so even when we get some research on what might be working for students or what they might need in the workplace, that was from a model that was a year ago, and now is it relevant? Now that will settle down eventually for that stuff, but I don’t see any short-term thing there. I think it’s more we have to understand that we’re running big experiments all the time now, and not only the students learning, but we are learning as well as we do them. A lot of faculty I’ve encountered have chosen: Well, what if we just pretend that the thing doesn’t exist?” [LAUGHTER] And the problem with that is you will not learn, you cannot learn, you cannot serve your students if you pretend the thing doesn’t exist. Now, there could be great rationales for trying to work out what you want the students to learn without AI and what you want them to learn with AI, and we’ve had this challenge before. We had the same challenge when we had calculators, and we had the same challenge when I was told, when I was going through high school, I wasn’t allowed to type up my assignments at all, because computers started to have spell checkers, and that boy did they not want me to use computers to spell correctly. I should just somehow do it myself, even though let’s face it, the second I was out of there, I never had to worry about that again. So [LAUGHTER] that’s the kind of things that we’re dealing with, but there are things you have to learn, and any new technology really sort of forces you to think about what is it that I need to get into a student’s head, and will AI get in the way of that, which it could.

Rebecca: Probably like the need to prioritize limited resources, limited time with students. You have to decide what’s the most important to focus on, and what maybe we can offset to technology.

Joshua:: Indeed.

Rebecca: It’s always about resource management, isn’t it? These economists….

Joshua:: Ah yeah, exactly, [LAUGHTER] that’s correct. [LAUGHTER]

John: Going back, though, to the labor force, as Rebecca had mentioned before, there’s a lot of students who are really anxious about what their job prospects are like, and we’ve seen in the last couple of years that the job prospects in the U.S., at least, for a new college graduate have deteriorated quite a bit, and it’s not clear how much of that is due to AI, how much, because of the random policy shifts that are occurring at the federal level, or whatever else might be causing these things. One concern that’s coming up a lot in discussions is whether college is still going to be relevant in a world in which people have access to AI tools, and can use them to substitute for the types of learning that occurs in college. Do you think college is going to disappear or change fundamentally in some way?

Joshua:: Well, that is a, of course, a great question, but it’s not a question that just exists because of AI. I think, actually, with proper introspection, we should wonder about whether we have been, for the past few decades, serving students well. Now the evidence is that they earn more, they can get more jobs and other things for a college degree than with not. In the U.S., at least they’re quite expensive. It’s not clear what the value of that is, since at other places it’s much cheaper and students seem to get similar benefits. So we could discuss that. So there’s always a constant problem of what is relevant, what is out of college, and also half of what you get out of college is not related to the formal education, and it’s related to other things, growing up and socializing, and becoming an adult, and all those other things we list [LAUGHTER] to justify our existence. But the student anxiety and stuff, I’m not surprised, students are anxious about AI, because all of the big proponents of AI are going running around saying they’re going to wipe out all the jobs, and I have never bought into that. I’m, amongst my colleagues, a very prolific user of AI. I use it for all manner of things, and some of those things are quite excellent. I’m a head of department, and I use AI now to get rid of all of those reports and things that I have to write for the university that might have taken a day. They now get done in seconds, and it’s great because no one was ever going to read those reports. They just wanted to see that they existed, and now that I can use AI, they exist, much like students producing assignments, and so I can’t blame them. Students are producing assignments, and there’s producing for professors who, for the most part, don’t want to actually read them. Anyway, nonetheless, [LAUGHTER] the thing that I’ve seen from that is that anything that is enjoyable and similarly meaningful to you, and also involves judgment and taste, these are things that you can’t get from the AI. These AI models, as we wrote in our first book, are prediction machines. They’re good at certain aspects of intelligence, but not others. They’re quite miraculous in the way they’ve evolved over the time I’ve been studying them. No doubt about that. They’re amazing things, but in all of it, I haven’t had any reason to doubt the value and usefulness of what humans do, even though it’s sometimes very difficult to define. It’s come in my own experience, it’s come in all of the data that is generated. As I said, we can’t identify whether people have been losing jobs from AI. Will some people lose jobs from AI? No doubt, because it’s going to reorganize things, that happens, but is there going to be the mass 50% unemployment that somehow engineers in Silicon Valley can imagine, but no one else can. That’s not going to happen. And fundamentally, it’s this: if you believe enough that humans matter, that even with the best AI, humans provide something of value. If you believe that, you can simultaneously believe that AI is going to be great, and AI is going to be great for people. It’s only where you are in a mindset that I think is what drove the Silicon Valley types, is that humans are crappy, stupid, and should be replaced, that you get big use of AI and big displacement of humans. And so, fundamentally, that’s the different assumptions, people who are anxious have to say: “Is there something… what is it that I can do?” Now, I mean, the problem is that, being older folk, we sort of have more opinions about our value [LAUGHTER] as we’ve learned it over time, and university graduates, all they’ve been doing is not getting that, [LAUGHTER] not yet. And so it’s sort of like you would have to take that on faith, but I must admit I’ve seen no evidence to the contrary. That’s my resident belief at the moment, and so I’m on team human.

Rebecca: So pro human, I hear.

Joshua:: And I was perfectly open to the idea that we weren’t. I mean, when it comes down to it, my use of AI has been a whole lot of things that I’ve decided the AI is better than I am. I’ve decided that literally answering student questions, I think it’s better than I am… [LAUGHTER] doing those sorts of things. But ther’re other things, there’s enough of the other stuff, enough of the other stuff that I think it works, and I can spend more time doing that, so even better.

John: And at two in the morning, AI tends to be much better at answering student questions than we do.

Joshua:: Exactly, and the students know that too. The students don’t like going… there’s some students who like seeing professors, students who are really at the top of the class want to see it, but not the seventy on the test. And then the students who are struggling also like to go because they need that extra help. And the beauty of it is, is if you use All Day TA, you don’t have the mass in the middle coming and bothering you, and you can concentrate on the parts where you can actually provide the most value. It’s absolutely true. What do we get most value out of a professor? It’s engaging with really smart students who we think are going to go somewhere, or a struggling student who really was close to giving up, that you sort of explain things to, and they’ve been able to come through. Those are our rewarding parts. That’s what we want to do. We don’t like the students to come in the middle saying, “Will this be on the test? They can ask the AI.

John: And there is a fair amount of research that suggests that first-generation students and students with imposter syndrome are very unlikely to go to their professor to ask questions, because that would reveal that they don’t understand things that well, and they’d rather keep that hidden, and many of them would just give up. So having access to free tutoring service without any judgment 24 hours a day is a valuable service.

Joshua:: I would go beyond that. Well, I’m in one of the elite universities, University of Toronto. The vast majority of tertiary institutions are not that, they are ones where a lot of students struggling financially, they have other jobs, they’re working part time, they’re studying part time, and things like that, and those are the students who drop out. And why do you imagine they drop out? Because they’ve got these assessments, because they have under-resourced teachers as well. So, they have these high-stakes assessments, and it’s basically on the night before, they haven’t been able to learn it, and they can’t get any help. And they said, “Well, I can’t do this anymore.” Now, if just a few of those students, [LAUGHTER] because of All Day TA, can get the answers they need and be able to hump over those moments, they can complete their degree. And so, if we can just move a few percentage points on dropout rates, I will be more than thrilled.

Rebecca: Well, we always wrap up by asking, what’s next?

Joshua:: Well, what’s next? It’s very interesting to think about what’s next, because normally when we talk about what’s next, we’re thinking like in five or 10 years. Now I’m talking about what’s next is maybe later this year… [LAUGHTER] would be the sorts of things we’re thinking of, and I actually no longer have a very clear view of it. I kind of feel, “Oh, it’ll be exciting and new,” and I get excited anytime some new thing comes forward. I think what we’re going to see, what’s next, more is that the AI is going to start coming into the fore in the other parts of our jobs. As I already said, it can help administration, it can help teaching. We kinda know those. I think in research it’s going to be a very interesting development, where we will be confronted with our desire to know what the answer is versus what used to be a complimentary desire to be the one who found it, [LAUGHTER] and so we will have to reckon with that in regard, and everybody will have sort of views to it on that, but it’s not until it actually happens, or it happens to you that you really know how you feel about.

Rebecca: Much to look forward to, that’s for sure.

Joshua:: It’s going to be exciting. [LAUGHTER]

John: It is, and the rate of evolution of AI tools has been phenomenal. I don’t recall seeing anything else develop as rapidly as this.

Joshua:: No, it’s crazy. It is crazy, and I was studying this seven years before it occurred, [LAUGHTER] all the latest stuff, and it was like a surprise, how quickly it happened.

John: And just a few years ago, there were all these reports about people saying AI will be there in the next five years, and that kept getting extended, and all of a sudden it was there.

Joshua:: There it is. We arrived through the Turing test, the whole thing. Amazing.

John: Well, thank you. This was a really interesting conversation, and it’s a really interesting time to experience these things.

Joshua:: Thanks for having me. I’ve enjoyed it greatly.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

450. Collaborative AI Guidance

With the rise of AI, institutions are scrambling to provide guidance and policies for students and faculty. In this episode, Kathleen Landy and Joshua Foster join us to discuss a collaborative approach involving student government, administration, and faculty.

Kathleen is the Provost and Vice President for Academic Affairs at Genesee Community College. Prior to this, Kathleen has served in leadership roles at teaching centers at Cornell University, Queensborough Community College, and Mercy College. She has taught in multiple modalities and also has extensive K-12 teaching experience. Kathleen also served on the Pod Network Core Committee from 2022 through 2025 and currently serves as a co-chair of the SUNY Council on Assessment. Joshua is the Vice President for Academic Affairs for the Student Government Association at Genesee Community College.

Show Notes

Transcript

John: With the rise of AI, institutions are scrambling to provide guidance and policies for students and faculty. In this episode, we discuss a collaborative approach involving student government, administration, and faculty.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Kathleen Landy and Joshua Foster. Kathleen is the Provost and Vice President for Academic Affairs at Genesee Community College. Prior to this, Kathleen has served in leadership roles at teaching centers at Cornell University, Queensborough Community College, and Mercy College. She has taught in multiple modalities and also has extensive K-12 teaching experience. Kathleen also served on the Pod Network Core Committee from 2022 through 2025 and currently serves as a co-chair of the SUNY Council on Assessment. Joshua is the Vice President for Academic Affairs for the Student Government Association at Genesee Community College. Welcome, Joshua, and welcome back, Kathleen.

Kathleen: Thank you so much. Great to see you again.

Joshua: Thank you for having me here.

John: Thanks to both of you for joining.

Rebecca: Today’s teas are:… Joshua, are you drinking tea with us today?

Joshua: I am. I am having some Japanese Sencha green tea here in my GCC mug.

Rebecca: Amazing. And how about you, Kathleen?

Kathleen: Well, I don’t know if this is like wearing the concert T-shirt to the concert, like the band T-shirt to the concert, but I have a Tea for Teaching mug, and I’m defaulting to Irish breakfast tea.

Rebecca: Clearly one of the cool kids. [LAUGHTER]

Kathleen: That’s right, that’s right. [LAUGHTER]

Rebecca: How about you, John?

John: I have a Twinings English Breakfast tea today.

Rebecca: Oh, nice.

Kathleen: Very lovely.

Rebecca: I think I have… maybe I should verify…

John: Another imperial tea?

Rebecca: I do, [LAUGHTER] I have a Scottish breakfast tea today.

Kathleen: That sounds lovely. I don’t think I’ve ever had a Scottish breakfast tea.

Rebecca: It’s got a little more oomph to it. [LAUGHTER]

Kathleen: Okay.

Rebecca: A little bolder, I feel like.

Kathleen: There you go. Okay, I’ll have to check it out.

John: This is a follow-up to our earlier podcast, where we talked a little bit about program- level AI policies, and you mentioned that you had been working with a member of the Student Government Association to come up with some recommendations for faculty and students. Could you tell us a little bit about the origin of this collaboration on the policy?

Joshua: Sure, I would be happy to do that. As a liaison between students and faculty on campus, students often approach me with various concerns, and one concern that came up multiple times during the fall semester was students having their legitimate work flagged as partially or even completely AI-generated. This initially prompted a discussion in one of my classes, at which time I learned that many students actually avoid the use of, or even the mention of, AI in order to avoid the potential of a suspected academic integrity violation, but I also learned how frequently academic integrity violations related to AI do occur. So I realized this was a topic that necessitated a more direct and open conversation involving both the faculty and the student body, and that is what led Kathleen and I to develop these guidelines as a way to guide those conversations,

Kathleen: I’m happy to add to that, Joshua, because this came up organically in one of our routine meetings. So, Joshua and I, as part of his role at the SGA (Student Government Association), and my role in Academic Affairs, we meet every three weeks, and so this came up in one of our conversations, and it was so timely, and it was totally a function of happenstance that at the time I had been having conversations with academic program directors and multiple faculty about the necessity of having course-level AI policies, and so it was very serendipitous, almost, that there was this student concern that Joshua articulated, and I had been exploring this question with faculty, and it seemed like a really lovely opportunity to bring everyone together and explore this topic.,

Joshua: And I had actually heard at the time members of the faculty discussing that they did have concerns as well, and so it sounds like the people who we serve both had concerns, and so we could kind of bring those concerns together, merge them together in the development of these guidelines.

Rebecca: Can you talk a little bit about how you facilitated the creation of the guidelines? Who participated in developing this initial round and this initial draft?

Kathleen: So, Joshua and I, we opened a shared Word document, and we just got some thoughts together, and we thought initially it was a no-brainer to have this set of guidelines have two parts, one that was faculty facing and one that was student facing, and we knew that they would kind of run parallel to each other and reinforce each other, but we started with a little bit of a brain dump, and we just kind of mapped what we thought were good ideas and good starting points and points of entry for the conversations about AI and academic integrity, and we landed on, I guess, we would call a sacrificial draft that we thought we could both bring to our respective colleagues and get more input. So it was with real humility and curiosity that we did a little road tour. We hit the ground running with this document that we’ve shared with you as well, and I personally brought it to colleagues in Academic Senate. We have a standards committee, and so I brought it to the chairs of that committee and shared it with our academic program directors, and we went from there. We solicited feedback, and it was great feedback that we got. And Joshua, I don’t know if you want to speak to folks that you brought it to. I don’t know if it was the SGA cabinet.

Joshua: Yes, so I initially didn’t bring it to too many students. What I did rather was I sort of took what the guidelines were without actually taking the draft, and I took that and I talked to different stakeholders about their concerns, and I brought up where different points that were covered in those draft guidelines and got their feedback on each point that I thought was relevant to them, and then I would take that and contribute it back in, and then I would also look at what some of the different faculty members are who I talk with and get their feedback and kind of combine that feedback with the feedback from the students and bring that back to Kathleen and to her colleagues, and I think that was really a good collaborative way that we took this, to include both levels of feedback, but not have the students edit the guidelines, but have them indirectly contribute through my contribution, I think, allowed them to really be able to express their thoughts and their concerns more clearly.

John: We have a student committee on learning and teaching here, and this came up in all of our discussions with that group, and the same sort of concerns that you mentioned, Joshua, appeared, that some students were really upset that they were accused falsely of engaging in academic integrity violations, and also others were concerned about whether it was safe to use it, because they’d like to be able to develop skills in this, because it may be relevant for their careers. It’s a topic that I think applies to a lot of people. Could you give us an overview of the guidelines, both for faculty and for students in this draft document?

Kathleen: The guidelines are surprisingly simple, but we strove for them to be very clear. So, the guidelines include having a clear AI policy that’s articulated in every syllabus, even though that’s a soft expectation, we wanted to make it a little bit more concrete and a little bit more pronounced, and so we wanted to encourage faculty to have a clear policy that includes specific examples of acceptable and less acceptable uses of generative AI, and a disclaimer or an explanation about any possible AI detection platforms that might be in use for the course. So that’s kind of an equity issue. It’s a social justice, it’s a transparency related issue that if there are detection platforms in play, students should be aware of that. That should be a known element in any learning environment, and importantly, this same AI policy should have a clear list of consequences for initial and possibly for subsequent violations of the class policy. So we wanted to start with having something concrete in place, something that’s published for students and faculty alike to refer to. And then the second part of the guidelines for faculty involve posting open and candid conversations at the beginning of any course or assignment, something that came up in the previous discussion that we had was this idea that in sympathy for the schedules that students have and the potential for them having five different course level policies to keep track of, it’s really not realistic for any given course level policy to be expressed once and then never mentioned again, or to be published on page eight of the syllabus, and then never revisited again. So, the idea of not only having a clear AI policy, but revisiting that and explaining it and communicating it frequently and clearly, that’s a kindness that we can show to students. It’s a good practice. So, this idea of hosting ongoing and candid conversations within the class, especially at the beginning of a course or the beginning of an assignment, that’s really critical. And then finally, the guidelines involve one or two strategies for fostering greater student buy-in, for lack of a better term, or a greater sense of accountability among students for the policy, so things like having students sign an integrity statement, a commitment to acting within the confines of an academic integrity policy, having students sign an integrity statement, or including a question about AI use and implications within the context of a syllabus quiz, or something that’s a little bit more interactive. We thought that would allow for a structuring of engagement with the policy, to not just have it live somewhere, but to have students actually engage with the policy.

Rebecca: It seems really important not to just have something exist on paper or some virtual ether, right? [LAUGHTER] It’s like have people actually interact with it.

Kathleen: That’s right, especially because for many students, and this will change over time, of course, but they might not be accustomed to having to check with the course-level policy. These are relatively new and emerging caveats or elements of a syllabus. So, how could we not redirect students to say, “Hey, remember we have this policy, we have this expectation?” It’s really a kindness, I think, to students. And then Joshua, I’m happy to turn it to you for the student side.

Joshua: Yeah, thank you, Kathleen. So, the guidelines for students really do mirror and complement the guidelines for faculty, and I want to start by pointing out that, similarly to the guidelines for faculty, the guidelines for students do not dictate when the use of AI is appropriate or not appropriate. Instead, the goal of the guidelines is to help students learn how to communicate with their instructors to be sure that the course policies are clearly understood prior to any misunderstandings happening. So, with that in mind, the guidelines mentioned topics like requesting a course policy on AI, if one is not provided, and asking questions of the professor or the instructor to understand that policy. They encourage attending office hours, sending a professor an email with questions, or even asking a question in class, so that other students can benefit. Maybe other students have that same question, and maybe they don’t want to ask for the reasons that we discussed earlier. So, the guidelines, in addition to this, include steps for how a student could start a conversation with a faculty member if that student’s work has been flagged as having been AI generated, regardless of whether or not that work was correctly flagged as having been AI generated. The guidelines stress being respectful and aware that misunderstandings do occur, but they also encourage students to advocate for themselves when necessary, and I personally believe that student self-advocacy is a very important skill for students to learn early on during their college experience. The last major focus of the guidelines for students is the importance of building strong and collegial relationships with faculty members, which can be difficult or is something that is even avoided by some students, especially younger or first- nyear college students who are still adjusting to leaving a high school environment, and I believe that this is actually one of the most important parts of the guidelines.

Rebecca: I think I hear some really common themes around transparency and relationship building.

Kathleen: There’s no doubt. I was speaking with Joshua. We had a debrief of an event that we co-facilitated this past Friday, and I was struck by the extent to which the following statement is true. Almost all of my recommendations or strategies or suggestions for how to handle issues involving academic integrity and the use of AI, in particular, almost all of my strategies involve the human elements in a learning environment, and they’re all relational. It all comes down to either communicate… not all, but it often comes down to communication and relationships rather than the tech itself, and so I think that’s a really interesting relationship: that the tech challenges that we encounter often have very human or relationship-centered solutions. That’s been my experience to date. So I agree with you, Rebecca. There are trends in the solutions that we keep coming to.

John: I believe you also did some sessions with faculty, staff, and students about this. Could you tell us a little bit about that, and what sort of reactions were received?

Kathleen: So, we’ve had one session, and we hope it will be the first of many, and I have to tell you, it was awesome. We were delighted, we had over 50 participants on a Friday afternoon, no less, which is no small feat. We were very excited by the turnout, and we tried to be very mindful of how we engaged folks from the beginning. We wanted to set the right tone, and so we started with a little bit of a series of short interactive polls to let folks know that we’re all coming at this from a point of intellectual curiosity, and we wanted to build a little bit of a sense of community around this topic, and so the very simple question that we started with, it was a simple qualitative poll everywhere to just text your answer in, and we asked a very simple question. We asked what word best captures how you feel about generative AI in your classes, very simple, and we knew we could get anything, and so we gave everyone simple directions. Everyone had the option to pass if they wanted to, but everyone had the option to text a single word, and we generated a word cloud right there. I can send you, if you like, a picture of this word cloud. I’m looking right at it. I have to say, it was really lovely to see the extent to which students could see the faculty’s human responses to this question, and faculty could see students’ human responses to this question, and so the responses were anonymous, but the word cloud was really thought-provoking, and it was populated with words like conflicted, concerned, curious, complicated, stressful, uneasy, overwhelmed. I think it set a tone where everyone could kind of check their egos at the door, or check their personas of being an academic, or a scholar, or an honor student, what have you, fill in the blank. Everyone could check that at the door, and we all kind of became students of this together, and it was, I thought, surprisingly effective at just setting a nice tone of inquiry for the remainder of the session. The session was about an hour long, and so we had interactive polls and a bit of an overview of the guidelines. We distributed a physical copy, like a paper copy of these brief guidelines in the room, and we posted them for the hybrid. It was a hybrid session, that’s worth noting. We posted them for the online participants, and it generated rich discussion. We got some great suggestions from both instructors and staff alike. We had students there who weighed in. We had folks, both students and instructors, share experiences, times where they wish they had these guidelines. Some expressed a concern about having or desiring more robust guidelines, like they wanted more of the same. And I’ll maybe stop there, because Joshua, I’m sure you want to add to that, but I was surprised at how eager folks were, and I was gratified by the enthusiasm that the group expressed.

Joshua: I was very excited as well, after having been at that session, and I joked with Kathleen a little bit at our follow-up that I wish we had had two or three more hours to go on, because I think that we certainly could have, but I think that the thing I was most pleased about was that we had a good amount of both faculty and students in attendance, and that they could see that, in most cases, they are in agreement with each other on these issues, or at least there is some mutual common ground between our students and our faculty, and I think that was the first thing that was really important for both of those groups to see, and then the second thing was that a lot of students came to me afterwards and said that they were really excited to have learned about how AI is becoming a common issue and how interested people are in it. As an accounting student, I have found that within my classes I’ve had a lot of people who are both interested in AI and who are both very hesitant about it, and ironically, if I had to make a really, really wide swoop with a brush to say who is more pro the implementation of AI, at least in the specific classes that I have been in, it’s the faculty, not the students, and so I think that the students being able to see that was really helpful, that that may be what we students imagine that the faculty are thinking about, and what the faculty maybe imagine that the students are thinking about maybe those are being too heavily stereotyped, maybe those aren’t really, really how we are seeing this, and so I think that was really good to have experienced, and I just think it was the meeting was very well received. There was obviously a lot of desire for follow up to it, for more conversations, but I think that the conversation that we did have was a very good starting point, and it helped to really demonstrate that where the direction that we’re going is really the direction that we should be going.

Kathleen: Thanks, Joshua. The only thing I would add, because I don’t want to paint this picture that it’s all rainbows, and we all sang together at the end of this session. I did have a couple of faculty members, and I was super grateful that they attended. They didn’t express frustration, per se, but their response was more like, “yeah, and…” like they were just ready, and I’m loosely paraphrasing, but this is close to the exact language. They said, “this is good, this is fine, Kathleen, but this isn’t the real work. The real work has to come in the articulation of departmental policies, and what’s that look like?” And I guess my response to that was, “well, yeah, of course, that’s the departmental next step, and that’s not for Academic Affairs or SGA, for that matter, to dictate to departments what their departmental or course level policy should be,” and so I think the workshop work is what should follow. So, there was a little bit of maybe frustration, that might be too charged of a word, but there was a little bit of feedback that was, “yeah, and okay, like, get to it, get to the next step,” which indirectly I interpret as affirming feedback that we’re on the right track, that folks are eager for guidance or at least engagement on this topic.

Joshua: And I believe that the same is true for students. I think that students really want that greater level of course-level policy or program-level policy, but I think that we’re not to that point yet, and so at this point a student came to me today and asked me. My response would be, “think about communication, because that’s what we are able to do now. That’s what we can do now.” Those other more clear things will come, but now we have to focus on what we can realistically do together.

Rebecca: It seems like a really valuable opportunity to demystify what people are thinking about, and that’s a really important first step into moving towards policy, or something that’s more formalized. You’ve certainly presented this as a working draft, and some revisions, and things like that. What do you see as the next steps to maybe formalizing this work at an institutional level, and then obviously there’s some work that would need to follow at the departmental level.

Kathleen:I think there’s a few next steps, and one has to deal with instructional practice and faculty development, and the other has to deal with steps to take leading towards policy articulation. I think the immediate next step involves endorsement, to take the feedback, the great feedback we’ve been given, and refine these guidelines, share them again, probably have a few more rounds to get some greater input, but then go through the formal channels of shared governance to have endorsements, formal endorsements, so that we can say “here are some endorsed guidelines, and those endorsements are coming from Academic Senate, from our local union, from Cabinet, from SGA, from program chairs.” We can get a lot of structures on board to kind of promote these and broadly share them. On the instructional side of things, and this is connected closely, of course, to faculty development, I think we can generate conversations about all the things that we’ve been discussing and all the things that are focused on in these guidelines. One of the polls that we asked, one of the questions we asked this group, and we had some questions restricted to students and some restricted to faculty in this one- hour session that we had, and one of the faculty questions was very simple, yes or no question: “Do you presently have a clear published AI policy for your courses?” And 57% of faculty said yes, and clearly 43%, my follow-up question is, “why not?” Let’s get that number a little higher of those that do have a clear policy, and I think that’s really good for that 43% of faculty to see, like, “oh, wow, 57% of my colleagues present at this session have something in place. Let’s talk about what that is, and maybe I should have that,” and I think that those collegial conversations, and then doing the work of adopting, crafting your own AI policy, and publishing that in the context of whether it’s a Brightspace shell or a syllabus, likely multiple places, that’s part of what’s involved in the next steps from my perspective.

John:I know in our campus there’s been a fairly wide range of perceptions concerning AI detection software. Some faculty trust them very strongly, even when the evidence suggests otherwise. Other faculty very reluctant to rely on them at all, except maybe as a signal that they may want to contact a student to talk about the issue. I was really pleased to see that your guidelines explicitly stated that AI detection software is fallible. Could you talk a little bit about that, and whether people in general have accepted that message there?

Kathleen: I’d like to think they have accepted it, I think, and I also think it’s a case that folks are still exploring what detection platforms do and how they operate, and how they might be of use to them in their own classes. So, I do think it’s variable in terms of comfort with and knowledge of these platforms, but I think there’s a lot of false positives, there’s a lot of false detections, and I think disclaiming that and modeling how to have the conversations about that, that’s something that we hoped the guidelines would get into. I think dragging that issue into the light is part of the intent of these sessions. I think folks are pretty receptive to it, because there might even be a self-consciousness about what we don’t yet know. So once again, I do think there’s a dotted line to faculty development there, and I think we have to model the questions that we ask as we analyze the efficacy of these platforms. So I think it’s time for candor, and I think it’s a time for inquiry, and just the humble recognition that we have to talk about what we need, what we want, and how good these platforms are. How efficacious are they?

Joshua: And something that one of the faculty members, who was present at our session, brought up, and something that I really appreciated from the student perspective is the acknowledgement that because these AI detection platforms are not perfect, they can actually serve to erode the level of trust that has been built between students and faculty members. And so it’s unfortunate that they do that, but that’s one thing that will have to be accepted when using an AI detection platform, and so, as a student, I appreciate that faculty are acknowledging that reality, and so that means to me that they will be more sympathetic and more understanding, because obviously in some cases those tools, those positives, are accurate, but in other cases they may be a false positive. And so just having that awareness and the fact that faculty members are acknowledging that level of student trust that may be called into question, I think is really good for them to be able to recognize.

Kathleen: Well, it’s great that you say that, Joshua, and I do want to respond to it. It prompts me to underscore this. I think the disclaimers regarding the effectiveness of AI detection platforms, it drives the conversation to prevention rather than detection, and I think that’s a huge part of not only instructional and assessment practices, but also the learning environment, and it goes back to the relationships and the trust that you mentioned, Joshua. So, I think that’s a huge part of it, the shift from detection to prevention, that’s a nice point of entry for me, and the conversations with faculty.

Joshua: And I think that that level of trust between students and faculty may be the first step. probably is the first step, in that prevention.

Kathleen: That’s right. Agreed.

Rebecca: So, you’ve talked a little bit about wanting, or even departments, maybe, being eager for like the next steps. Can you talk a little bit about what movement there has been in terms of the course-level or program-level policies on AI, or what you maybe expect to see as we enter into the fall?

Kathleen: We’ve had a few departments, a few programs, or program clusters, I should say. We’ve had a few folks self-organize, and they’ve proven to be self-starters in this domain, and they’re really gathering and just meeting routinely to articulate policies and to explore what other institutions have to craft that together, and that’s inspiring. That’s really great to see. Our business department comes to mind immediately. They’re doing great work, and there are others that I think are curious, and folks are kind of coming to those realizations of, hey, we need to huddle up and figure this out. I think folks are coming to that conclusion on slightly different yet overlapping timelines. So, I think there’s a role for my role and Academic Affairs Administration to prompt that collaboration a bit more intentionally, whether it’s the articulation of a strategic objective relative to having a target, the number of academic programs to articulate as policy, regardless of what it is, I think we can prompt that a bit with the strategic planning that we do, with the faculty development planning that we do, and I think to a certain extent we’ve done that, but I think we can turn that up a little bit. I think we can be a bit more intentional about it, but it’s happening just at different degrees and varying by program.

John: We talked a little bit about AI detection and how perhaps that could be used to nudge people in ways to use AI within the guidelines of the course, but are there other strategies that faculty are using to encourage students to use AI ethically rather than to shortcut their learning?

Kathleen: It’s a great question. I think, and this will vary by discipline, so I don’t want to speak in too broad of terms here, but I might end up doing so. I think to the extent that AI can be leveraged to enhance or scaffold or facilitate a process, I think that’s always a good thing. And I think if the use of AI can underscore the importance of the integrity or ethics-related outcomes for a course or a discipline, that’s great. I think there’s incredible potential for use cases, in terms of here’s how to use, here’s how not to use, here’s how this could go really well, or here’s the quickest way to work yourself out of a position, a passion, a process, whatever is appropriate there. So, I think there are examples, but they’re highly discipline-specific, and I know that that’s not the most robust answer, but I do see evidence across disciplines that folks are incorporating AI in really transparent ways that scaffold learning rather than inhibit it.

John: Joshua, do you want to take a stab at that in terms of things that you’ve seen where faculty are encouraging students to use AI in productive ways,

Rebecca: …or even how students are doing that on their own. [LAUGHTER]

Kathleen: Yeah, yeah.

Joshua: True. Yes, I have seen it. So, Kathleen shared briefly earlier that one of our questions was how many faculty members have an AI course policy, but another one of our questions that was restricted to students was, “How many of you, how many students used AI to complete their coursework?” …and that’s not meant to be AI that’s sanctioned, that is meant to be using AI in appropriate ways, assumingly, and that answer was 89%. 89% answered yes. And this is an interesting thing, because in my personal experience, the people I’ve talked to, I would say that that percentage is probably lower, maybe more around 60% but I found that answer quite interesting, and from the students that I’ve talked to, even the faculty that I’ve talked to, or other tutors, people who work with students, a lot of students are using AI in ways, for example, they may ask for help coming up with a topic idea, or maybe to brainstorm a topic, or maybe they’re brainstorming a first paragraph, or a direction that they could go within an assignment, or maybe they’re comparing scenario, and I think that those things, those getting started is one way that I’ve seen a lot of my peers using AI. Another thing that I have seen is where they use it for research. Now that will get into another topic, which is the reliability of that research, and so that’s something that would be important to talk about, but I do see students using it. I also see a lot of students using it, and they have very good reasons for that. But I think that it is something that is becoming more in use. I’ve also seen faculty members, and not in particular any faculty members that I have personally had a class with, but who I have talked to, or have actually had assignments in which they ask students to use an AI tool to begin an assignment. For example, there was an assignment in one class that was used for the creation of a resume, and one of the parts of the assignment was to take the resume and run it through an AI tool and have it give you suggestions and feedback on where you could make some words stronger, and so I think that those are ways that AI could be used in ways that are actually helpful or productive for learning.

Kathleen: If I can add one additional comment, and Josh, your comments prompted this thought. One of the more valuable pieces of feedback that we received on the guidelines was our oversight or negligence in not mentioning the library and the learning center in the guidelines, and so to your question, John, I think it’s a really good opportunity, a great idea, in fact, to leverage the use of AI to facilitate engagement with the other support services and the library on our respective campuses. So I think to build that kind of analytical info literacy related analysis of AI generated content as part of a process for a course-based assignment, I think that kind of a no brainer that we have this fodder, this generated content that we can use to engage with tutors and librarians, and of course, faculty, TAs, fellow students to kind of engage in the analysis, the fact finding, the verification of information, all of the info lit and AI enhanced info lit competencies that we’re looking to develop. I think there’s opportunities there as well.

Joshua: The other thing that I have had some students tell me is that they turn to AI because they genuinely need help thinking through something, or they struggle with a certain topic. So, for example, I know a couple of students who use grammarly because they struggle with their grammar, and I think that one thing that we are encouraging faculty to do with these guidelines, and thus also encouraging students to do these guidelines, is to have faculty make clear when these tools are allowed, and have students understand when they’re not allowed, because sometimes there’s a fine line between when is something like Grammarly, when is that allowed, or when is it allowed to use Chat GPT to think through your thought versus when is that prohibited, and I think that having those guidelines is essential, including those in your course policy is essential. The other thing that Kathleen was mentioning with the library and with the tutoring, is those are alternatives, so when AI is not permitted, there are other things that can help students. They aren’t just left on their own. There are tutors. Tutors can help you think through your ideas. There are librarians. Librarians can help find information. They’re very good at information literacy and finding those information. They can also help to verify research, and maybe if research was done via an AI tool, maybe that information needs to be verified from another more academic source. And so librarians are very helpful with that as well.

Kathleen: Now, Joshua, I kind of want to jump in, and I want to challenge one phrase that you mentioned, and I’m doing so as your thought partner in this, so you’ll appreciate the sentiments and the collegial approach here, but I would not want to take away from your comments to be that librarians and the Learning Center are alternatives to generative AI. I think that’s a dangerous framing, and I know that you don’t mean to say that they’re like alternatives to AI, but I think that distinguishing between the tool and the resource that many AI platforms offer, distinguishing between that and the human-centered expert librarian and the qualified and well-situated, well-resourced tutor, they can help us use this tool to our advantage and help us navigate the best uses, for that’s often a distinction. I wouldn’t want any of our librarian and tutor colleagues to say, “Wait a minute,” of course, if you don’t mind me clarifying what I think I know you mean by that. I think that’s important.

Joshua: I don’t mind, and I think that there are going to be times, though, when students choose not to use AI, and when they choose that they would rather have that more experienced human they can go and talk to, and I think that’s what I was trying to say, is that you don’t have to use AI to succeed in your learning. There was so long that this wasn’t available, and so I think that it’s coming available, and it’s a great tool, but it’s not the be all end all. It’s becoming a great tool, but you still need to be able to learn as a student and be able to talk to people, and I think that is really, really important.

Kathleen: Yeah, and I think going back to the spirit of our session, we have to figure out what this means together, what AI use means in our respective disciplines and our respective classes, which is absolutely how we approached this set of conversations. So, I think it may be the be all end all, it may not be. I think we got to figure that out together, which is the spirit of these guidelines, certainly, and the conversations around them. It’s important.

Rebecca: And pointing students and faculty to all the available resources and the roles that they can play in learning is always really important.

Kathleen: Yes.

Rebecca: So we always wrap up by asking: “what’s next?”

Kathleen: In terms of what’s next. I think it’s a clear next step for us to continue bringing faculty and students together, that was so elucidating. It was really great for that exchange of perspectives and for that shared exploration of this really important and impactful topic. But I think we got to keep at it. We got to keep bringing people together in thoughtful ways. That’s what’s next from my perspective.

Joshua: Yeah, I agree with that, and I think whether it’s bringing students and faculty together again or talking among students, talking among faculty, and then coming back together, regardless of how it’s done, just continuing that conversation is really the next step. You know, I agree, Kathleen, having that continued conversation is just really important.

John: We know AI has been somewhat disruptive for students and for faculty, and it’s nice to see this sort of collaboration, and it’s also something that can take place really nicely at the class level as well as at the institutional level. And it sounds like you are making some really good progress there at GCC.

Kathleen: We’re really trying. Thanks so much to you both. Thank you.

Joshua: Thank you. It’s been wonderful to be here to talk with him today.

Rebecca: Yeah, it’s great to have both of your perspectives, because I think the model you’re sharing around the topic of AI could be applied to many other challenging topics that challenge both faculty and students, and bringing those two groups together to work through those challenges and establish guidelines and other strategies is really helpful.

Kathleen: Thank you. I think so too. One of the goals is to model. We’re trying to model as a campus community for each other, whether you’re a faculty, staff, student. We’re trying to model how to problem solve and how to figure things out together and do so in a way that’s ethically sound and mission-centered, and focused on learning. So, we’re trying, we’re hopeful, but we’ll get through it together.

John: And when both students and faculty are involved in making these policies and have a say, there’s going to be a lot more buy-in to the policies than if they were just imposed.

Kathleen: That’s right. We agree.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

444. Random Audits to Deter Cheating

The growing capabilities of generative AI platforms have made it increasingly difficult for faculty to reliably distinguish between student work and AI-generated output. In this episode, David Wiley joins us to discuss the possibility of using random audits to promote academic integrity in a scalable manner.

David has an extensive record as an innovator, entrepreneur, and leader in open educational materials beginning with the Open Content Project in 1998, continuing with his work as Director of Educational Licenses for Creative Commons, a co-founder of several education-related organizations, including Lumen Learning, where he also served as the Chief Academic Officer from 2012 to 2025. He is the recipient of numerous awards, including an NSF Career grant. David is currently an Associate Professor at Marshall University where he teaches courses in Entrepreneurship and Management Information Systems. Much of his recent work has been on the intersection of generative AI, open education, entrepreneurship, instructional design, and student success.

Show Notes

Transcript

John: The growing capabilities of generative AI platforms have made it increasingly difficult for faculty to reliably distinguish between student work and AI-generated output. In this episode, we discuss the possibility of using random audits to promote academic integrity in a scalable manner.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is David Wiley. David has an extensive record as an innovator, entrepreneur, and leader in open educational materials beginning with the Open Content Project in 1998, continuing with his work as Director of Educational Licenses for Creative Commons, a co-founder of several education-related organizations, including Lumen Learning, where he also served as the Chief Academic Officer from 2012 to 2025. He is the recipient of numerous awards, including an NSF Career grant. David is currently an Associate Professor at Marshall University where he teaches courses in Entrepreneurship and Management Information Systems. Much of his recent work has been on the intersection of generative AI, open education, entrepreneurship, instructional design, and student success. Welcome David.

David: Thank you. [LAUGHTER] I appreciate the invitation to be here.

John: Well, we’re very glad you accepted. Today’s teas are:.. David, are you drinking any tea with us today?

David: I’m a water guy. I’ve got a giant glass of water here. I will be drinking throughout. [LAUGHTER]

John: Hydration is healthy. And Rebecca?

Rebecca: I’ve got English Tea Time today, John.

John: And I have a black raspberry green tea today.

Rebecca: Oh, that sounds nice. I’m sure it smells nice, and I probably wouldn’t like the taste of it. [LAUGHTER]

John: It’s delicious, and it smells nice too.

Rebecca: We have an ongoing discussion of that. [LAUGHTER] David, we invited you here today to discuss your March 9, 2026, working paper entitled “Random Audits as a Scalable Deterrent to Cheating, Using Game Theory to Design Fair and Effective Academic Integrity Systems for the AI Era.” You begin this paper by noting that there has been a long standing arms race between students engaged in cheating and faculty attempting to detect and deter cheating through the use of surveillance technology such as remote proctoring AI detection software and lockdown browsers. Is this an arms race that faculty can win given [LAUGHTER] the vast amounts of funding going into the development of generative AI platforms?

David: I don’t know that it was an arms race they could win before generative AI became a thing. It seems like there’s boundless, infinite creativity and energy on the part of students in this regard, in particular. We tend to be little older, a little less connected to everything that’s going on currently, and the availability of new techniques and new tools, I don’t see any end to this arms race. If we’re trying to fight it, technology versus technology specifically, there’s nothing we’re ever going to create as instructors that’s going to compete, as you say, with these multi-billion dollar generative AI tools that are being created.

John: What you suggest is that we skip the arms race entirely by perhaps using an audit system that may alter students’ incentives. Could you give our listeners an overview of how an audit system might work?

David: Yeah, it might be easiest to begin explaining by reference to some examples that people already know. We often make progress on hard problems by realizing that a different framing of the problem can help us see it from another perspective. This is a problem that I’ve been interested in for a while. Other people have been more interested in it than me, I acknowledge that. This isn’t my area of expertise by any stretch. I was just trying to abstract this problem up to the highest level I possibly could, and think about it from that perspective, and I would say that from that very high level, there’s some largish number of people that we need to be honest, but the number of them is so large and the circumstances are such that we can’t directly observe all of them. If you frame it that way, and then start to think about, what are other domains where that same problem exists, and how do they approach it, then some other examples start to present themselves. Doing taxes is the first one that came to mind. The IRS has an interest in all of us being honest, but obviously can’t observe all of our documentation and all of the work we do as we prepare our taxes and submit. Once you get to that first example, then others kind of start busting through. You think about anti-doping in sports. We can’t watch everything that every athlete does all the time, but we need them to be honest about the fact that they’re not using substances that we don’t want them to use. Nuclear proliferation [LAUGHTER] would be another… food safety, we can’t be in every warehouse producing every kind of food, the FDA can’t all the time. We need them to be honest that they’re following safety guidelines and meeting standards. Once you abstract the problem up one level, you’re able to see that same problem manifest in other domains. And of course, the way that the problem is addressed in all those other domains is through some kind of audit procedure, ostensibly random audits that have pretty significant penalties associated with them. And the idea is that the penalty is so dramatic that it removes the incentive to be dishonest in the first place. As you said, rather than trying to continue to compete in the arms race with “you develop this technique for cheating, and I’ll develop this one for countering it, and we’ll just keep doing that over and over again,” what if we just step to the side and said, “Let’s just try to remove the incentives for cheating altogether and kind of come at this a different way.” That was the originating thought behind the paper.

Rebecca: So that’s like the abstract, the big problem. And I think when you start thinking about it with these other examples, it seems like it’s not such a big deal, not such a big problem. It’s solvable. Can you talk a little bit about what that might actually look like in a classroom setting? What would an audit system look like in terms of assessments in a class?

David: I didn’t go into this very much in the paper and said explicitly in the paper that I think that there’s a second set of work that needs to be done. If people believe that this higher level framework is interesting and appealing, there’s a lot of detailed work to be done then in describing exactly what the audit would look like. And you can imagine it looking very different from one discipline to another, and looking very different from one kind of work that was done to another kind of work. So if I’ve given an exam, and this was a multiple-choice exam, say, with 40 or 50 questions, you’re selected for an audit, then what that might look like is I might take some subset of the questions that you got right on the exam, and I might watch you retake some subset of the exam over again, because I don’t want to spend another hour or hour and a half with you doing this. I want to spend 15 or 20 minutes doing it, and rather than randomly selecting questions from the exam, because maybe you missed some of them on the exam, it would behoove me to choose the ones that you’d gotten right on the exam, and we could look and see if you’re able to get the correct answers again. Or if I’ve had you write an essay, I might pull out five to 10 vocabulary words from that essay that seem kind of interesting. I might ask you just to describe to me what those vocabulary words mean to you. Give me an overview of the argument that you created in the paper. It would take a different format depending on what kind of work the student had done, which of course is related to the discipline of the course. But the idea is that you’d want these to be doable in a pretty brief period of time, 15 or 20 minutes, because you’re going to need to do a number of them, and you would only do them for major assignments. I wouldn’t do this for every 10 or 15 point homework assignment in a course. I would do it for a 100-point midterm, 100-point final, or 100-point project. I think it would need to be used pretty judiciously, because it is going to take a fair amount of time.

John: A lot of people recently have been suggesting a movement towards oral exams, and that’s a much more extreme form of this. And what I really liked about your approach is that it is an oral exam, but it’s one given only to a small subset of the students, and it’s a shorter exam designed to deter cheating, rather than to completely eliminate any possibility for cheating. And it seems like this is much more scalable than many other approaches that people have suggested. One of the things you mentioned is the Ellsberg paradox, which suggests that people are more cautious when they’re facing uncertainty, when they don’t know the probabilities, for example, of being audited, then when they know the risk, when they know what the probabilities are. How can instructors use this paradox to also deter cheating, or to help deter cheating?

David: Yeah, I think there are a couple of examples from behavioral economics that we could leverage to our benefit here as the instructors, but I think this is one. There is a interactive simulator that you can play with that accompanies the paper, where you can turn the knobs up and down on how many students there are and how many points there are in the course, tell you things like what percentage of people you need to audit and how big the penalty needs to be in order for the game theory to all work out, for the incentive to kind of evaporate, to cheat. You need to know all those things as an instructor, [LAUGHTER] but what you need to say to the class is, some of you are going to get tapped on the shoulder after the midterm, and we’re going to do this kind of supplementary learning review together. But I think it’s just as easy as avoiding saying “it’s going to be exactly 15% of you,” and then I think it’s important to come back afterwards and say I enjoyed the conversations that I had with many of you in our supplemental learning review. Make sure that people know that it’s happening, that it’s occurring. Hopefully, some of the people who are selected and who do well will come out the other side and tell their friends, “hey, you know, I got chosen for one of these.” It’s important that the word gets around and that people believe that they really are happening. If the idea of the audit as something that’s really going to occur loses its credibility, then of course, it loses all of its value as a deterrent.

Rebecca: So speaking of the idea of credibility, how do we really make sure that a system like this maintains its credibility, other than being careful about how we present it to the class.

David: Say a little more about that.

Rebecca: If students say, “Oh, I was tapped on the shoulder,” then it seems like whatever [LAUGHTER] you were tapped on the shoulder to discuss, or whatever that supplemental learning instruction was would need to seem to be like a good follow up. And that people didn’t feel like it was threatening in some way. If someone has test anxiety or something like that, it’s set up in a way that feels trustworthy and safe and that it’s fair.

David: Being fair is extremely important. Let me come back to that in a second. First, I would say, I’m not a person who gets upset easily. Let’s just say, a time that I was very upset, that I recall very clearly, was as a graduate student about halfway through my first course on assessment design and learning how to create a measurement of learning that is going to have the psychometric properties that you want it to have. It’s going to be valid, it’s going to be reliable. And realizing that my entire life had turned on decisions that had been made about what I had and hadn’t learned by teachers in high school and instructors in my undergraduate career that had no idea what they were doing as they designed these assessments [LAUGHTER] , they had taken multiple choice exams when they were a student, so they created them when they became a professor, and, oh my gosh, I got so lit up about that. I promise we’re coming back around to your question. But one of the interesting phenomena that happened with the move to online learning in the 2000s as that really started to take off, was people started making claims about the degree of quality and rigor and the standards that a course would have to meet in order to be offered online. And it was always hysterical to me that if you had turned around and tried to apply any of those criteria to the face-to-face instruction that was happening, very little of the face-to-face instruction would have passed muster. So there’s this weird kind of double standard that we’re accepting of things being, I’m going to say poorly done, that’s stronger than I mean, but we’re accepting of things being poorly done as long as they’re poorly done the way they’ve always been poorly done. But if we’re going to do anything different, it has to meet this 18 times higher bar. The reason I bring that up is that I think it’s the same person who’s going to design the assessment for the supplemental learning measurement as designed the original assessment. And so the idea that it would be somehow unfair or tricky or difficult or something like that, it will probably still be kind of poorly designed, just like the original assessment that was taken maybe was because maybe that person did their PhD in something other than education and has never had a class on what valid, reliable assessment design looks like. I don’t want to get in a place where we try to hold this secondary assessment to a 10x higher bar than we hold the primary assessment. I think as long as the secondary assessment looks very much like the primary assessment. That’s really what we want. And I mentioned in the paper that you wouldn’t want to put yourself in a situation where maybe your primary assessment, the exam that everyone does, is measuring learning outcomes at very low Bloom levels, like remember and understand. And then you would turn around on the secondary assessment, and all the questions would be apply and synthesize, and then you say, “Oh my gosh, you failed the supplemental exam.” Well, it’s broadly on the same topic, but it’s not measuring the same learning outcomes that the original assignment did. That’s why I think it’s nice in cases where you have some kind of exam with multiple questions, whether they’re short answer or multiple choice or whatever, to take a subset of those questions that the student did well on, that’s ideal. If you can’t do that, then you can take parallel forms of them, where, if it’s a word problem, you change the names of the people and the number of apples, the same question, just changing some of the details about it. But it’s very important that that supplemental assessment be on the same topic, at the same level of difficulty, similar in every way that it can be to the primary one. There’s no assessment strategy you could take to scale across higher ed if it really depended on all those assessments being valid.

John: To reiterate your point, it’s not just a few people who haven’t had much training in assessment and reliability and validity. It’s probably 95% or more of those teaching in college, I would suspect. It is a major issue, but that’s not something we can all deal with here, other than doing our best to help everyone become better at these things. [LAUGHTER] Well, going back to that point, about not sharing, necessarily, the proportion of students who are going to go through the second stage, because when students don’t know the probabilities, they’re more likely to worry about the downside of that, about the possibility that they may be caught. There’s another behavioral economic principle known as loss aversion, which is when people are faced with a choice between a gain and a loss, a loss of equal value to a gain is weighted much more heavily. There’s a lot of research suggesting that people do pay much more attention to potential losses than they do to potential gains. And I think this sort of approach is one where that loss is the main cost that students potentially face, and that may also amplify the effect.

David: I agree completely. As I worked on this idea, the loss aversion piece of it didn’t occur to me initially. Once I started doing some of the math and realizing the penalty was going to have to be bigger than 100 points to make the math math, as people say these days, that it was going to need to be 50 points more, or 100 points more. You were going to need a zero on this assessment, plus another 50 point penalty against what? …just against your overall grade. At some point, looking at that, I realized, “Oh my gosh, and people feel losses about twice as strongly as they feel gains of the same size.” That’s another bit of BE working in our favor there, in terms of, again, just trying to not fight the battle of, you’re going to cheat and I’m going to catch you. I’m just going to take away the incentive for you to cheat in the first place, so that we don’t have to fight that fight later.

Rebecca: I can imagine that introducing a system like this into a class would be challenging, like any type of different grading mechanism [LAUGHTER] where students are used to a particular kind of assessment or particular way of teaching and learning that may be counter to [LAUGHTER] good practices, but we want to introduce something new to them, we have to be really careful and take the time to explain the why. I imagine that would be the case here too, is really taking the time to explain why there’s an audit system and what the purpose of that is, would be important to getting the students to buy in.

David: Absolutely, I think so. I’ve had some preliminary conversations with folks in the Office of Academic Integrity at my institution to talk about, “Does this seem like it’s compatible with the way things work, and at what point do I refer someone to your office as having violated their academic integrity,” as opposed to “I’m just dealing with it internally, with penalties that are clearly outlined in the syllabus.” And it’s not me saying, “Ooh, I have vibes that you’re cheating,” or “I read this paper and it didn’t sound like you,” so I’m going to accuse you of something, but it’s just a repeated measures. You took one test, you did very, very well. We turned around and did a second measure of the same thing, and it was really different, and that’s a problem. So I think laying all that out and being very honest about the fact that I don’t want to do all the work it’s going to take me to do to try to catch you when you cheat, it’s just too much time. It’s too much effort. If I’ve got a class of 50 students, some percentage of them score a C or better, which creates the pool of people that I’m going to draw from to do these follow up learning measures, if I have to do five of them on every major assessment, but it actually eliminates the problem, that’s much less time spent than I would have spent on all the other things that you do. It definitely will take some time. And now, as I’m answering that question, I realized I’ve lost the thread of what the question was. I’ve meandered a bit from what you asked me there. I apologize.

Rebecca: I mean, I was just really thinking about faculty who introduce concepts like ungrading that are just really unfamiliar, because it’s very different from the ways that they’re expecting the assessment or getting caught cheating… like what any of that looks like. So for making it look different, then it really requires some explanation up front.

David: This is something that, if you don’t use a syllabus quiz, you would want a syllabus quiz where people have to go through and answer four or five questions about “Yes, I looked at it, yes, I understand I’ll lose all the points on that assignment.”” Well, and how many additional points will you lose off of your overall grade? Will it be 25, 50. 75, 100? It’ll be 100 points more. You’d really want to require them to go through some exercise where they have to read the syllabus, they have to understand, they have to demonstrate that they understand. You’d want to go over it verbally, either in front of the classroom or on a video if you’re teaching online. It’s going to take several explanations. And then I would raise the issue again just before any assessment where it applied. So everybody remembers, back in September, we talked about this, and now here we are at the big midterm, and this applies. So let me explain again what that means. Some number of you are going to take a secondary assessment after we take this primary one, and if there’s a big difference between how you do on the secondary one and the first one, there’s a penalty, and this is what the penalty is like. And the reason I’m doing this is to remove all incentive for you to cheat at all, because the odds of you passing the class if you cheat and are caught here, are basically zero. Maybe if you got 100% on every other assignment, you could squeak out a C- that’ll let you count this course toward your graduation. It’s just too dangerous, so don’t do it. You’re going to get caught.

John: And maybe even a reminder of that at the top of the assessment might not be a bad strategy, too.[LAUGHTER] So it’s there explicitly in case students ignored all the information up to that point, which has been known to happen once or twice.

David: Actually, that’s a really interesting point. Maybe the first two questions on the assessment should be, is there some chance that I’ll be selected for a secondary experience, and if there’s a radical difference between my score here and there, in addition to getting a zero on this, how many other points will I lose? Now, I wouldn’t want to put them in a state of panic that’s going to prevent them from performing well on the assessment, but you’re right, the proximity of the reminder to the task really ends up being important, doesn’t it?

Rebecca: If you really don’t know this, you should just let me know now. [LAUGHTER]

John: One of the other things you talk about, though, is a concern for false positives on this second-stage assessment, because students may be nervous, they may be having a bad day, there may be some anxiety or language issues or some other issue that interferes with their performance. So you suggest a few ways of perhaps addressing the risk of false positives. Can you talk a little bit about those?

David: Yeah, well, we’ve talked a little bit about one of them, which actually may not be in the paper, that is just been part of my thinking is that’s evolving, and that is using subsets of the same work that you had students do on the primary measure on the secondary measure, so that you remove any, what we might call construct irrelevant noise from that second measure. We want them to be the same across there. But if a student performed really poorly on the supplemental measure, they might come back and say a couple of things: “I was having a really bad day.” “My grandmother died.” “My cat died.” “This happened, this other thing happened, and I was just out of it.” Then I’d be inclined to say, “go home, sleep it off, eat a decent breakfast, and let’s try again tomorrow.” Now, if there’s something structural about the assessment that makes it difficult for them to engage with it, a) I would have expected that to show up the first time they took the assessment, b) any kind of accommodation that they require for anything that’s going on with them and their learning, we should have provided the first time around, and we should definitely provide the second time around. I think it’s important that there’s some kind of release valve for somebody who just wants to bang the pulpit and insist, “no, I do actually know this. There’s something wrong with the supplemental measure that you did” to say, “heck, let’s do it a third time. But this is the last one. With repeated measures. If I get two of one outcome and one out of the other, I know which one I’m inclined to believe now,” and when they’re quite different than that’s kind of concerning. I think the primary method is having there be some appeal process where they can take a second supplementary learning evaluation and maybe modifying the mode or the method of that. If there is something about the first one that if they’re English as a second language learner, and there’s a lot of writing on this version of it, then maybe there’s some alternate form that I can provide them that’s going to make it easier for them to show me what they really know, because the point here isn’t to catch anyone. The point is to get them to demonstrate what they actually know. So it’s not about gotchas. It’s about accurately measuring what students know and are able to do. So if there’s something about that secondary assessment that made it so that that measure was inaccurate, then we should take another swing at, but it’s not turtles all the way down, right? It’s not the 15th time that we’re coming around and trying this thing again. There has to be a stopping point.

Rebecca: Some students might not be concerned with maximizing [LAUGHTER] their grades. They might not be aiming for an A, perhaps they’re using AI to meet those minimum passing standards. Students say things like, “Cs, get you degrees.” If this is a common phenomenon, which I’m pretty sure it is, how could the audit process be modified to deter kind of that C level cheating?

David: One of the differences between the first draft of the paper and the draft that was posted has to do with this issue, and that is that I was trying to think about how to make the potential pool of people that might be drawn, that might be audited, smaller. And it occurred to me that if you failed the exam, there’s no need for me to audit you, because you failed the exam. Depending on the institution that you’re at and the discipline and the major, sometimes you need a C or better to count that course toward graduation requirements, sometimes just anything that passes is okay, but I think that as an instructor, you’d need to be kind of attuned to that, and that the floor, in other words, the lowest score with which someone would be eligible for the audit, needs to be far enough down that it picks up all the people in that category. So if it’s A, B, or C counts as satisfactory academic progress, then everybody who gets a C needs to be in the audit pool too, and they need to be drawn out, and they need to have the supplementary experience. If they knew it so poorly that even using AI, they could only pull out a C, then I think that’s going to show up, now if they’re being super clever and smart and trying to calibrate exactly their AI use so that they only get a C. Actually, in that case, they’re only cheating themselves. I mean, if they’re going to use AI, they should have just gone ahead and got an A, using it in a clever way to get a C doesn’t help you, because if you get pulled into the audit and it turns out you don’t know the material, you may as well have rolled the dice on getting an A than getting a C.

Rebecca: How to play the game well.

David: Well, this is part of why you don’t make all the rules explicit, right? It’s 12 and a half percent of you that are going to be audited every time, or something like that. I think to the extent that the rules are unclear, it’s harder to play the game.

John: Going back a little bit to addressing those false positives, you also mentioned perhaps having some type of an audit bonus or a bonus for those who are audited and passed successfully. Can you talk a little bit about why you might use that approach?

David: Rereading the paper before we got together today, I can see me disagreeing with myself a little bit in the paper [LAUGHTER] on this topic, which is why it’s just a working paper still. My initial inclination there was that if you’re going to get pulled out and have to schedule time and be at a place at a time to meet with me for 15 or 20 minutes to do something, you probably deserve something for doing that, something other than “attaboy” or “attagirl.” So introducing a couple of points that you could think of as extra credit or as bonus points, or as thanks for being part of this process, was the way I originally thought about that. It turns out that that bonus also helps the math work out in terms of the expected amount of points somebody could earn if there’s a chance of a false positive. But then if you don’t get a false positive, and you are able to show what you really knew, then you do get a couple of extra bonus points, like making that all balanced out, that it turns out to work out nicely from that perspective as well. Later in the paper, you can see me kind of arguing with myself about whether I want to call this extra credit or bonus points or, like, “I put you through this whole thing, guys, I’m going to throw you a bone. Here’s a couple of points for participating in this process,” versus saying “you took this much smaller second assessment, and since it was smaller, I’m going to give you a smaller amount of extra points for passing it, because you get points for passing the first one, get more points for passing the second one.” I’m still not exactly sure how to frame the language around it, whether to think about it as extra credit, or look, you have a chance to do a second assessment, and you’re going to earn more points on that, and those points are also going to count toward your grade, and other people aren’t going to get the opportunity to take that second one and not have the opportunity to earn those extra points. So you can think about it as being earned or being extra, or however you want to think about it, you can really only understand by putting this in front of students and working through it a time or three. There’s got to be some way to talk about it, though.

Rebecca: Like thinking through a student lens. “Ooh, I want to get audited so I can get some extra points.” [LAUGHTER]

John: If only the IRS worked that way.

Rebecca: I was just thinking. But I’m like in all these other contexts that doesn’t play out.

David: That does not work that way in any of the other contexts. And maybe this is just me being a softie, especially in an asynchronous, online course, if I’m going to require you to make an appointment and show up someplace at a certain time, I think you deserve something.

Rebecca: I think that feeds into the question of just in what ways is the system equitable, if we think about the student populations and thinking about your sampling and different strategies that might be employed. How do we make sure that a process like this remains equitable?

David: I think it being truly random is important to being equitable. And there’s a paragraph or two in the paper, some people who did poorly on the first two or three assessments, and now they suddenly do really well on this one, maybe we might weight their potential to be drawn out of the pool a little more heavily. You could certainly do something like that, but it seems like the most fair way to do it is just to do it completely randomly. And I think part of the equity argument here is that it’s really only fair if the grade that you earn is an accurate reflection of your learning. And if you work and study and push and strive and you end up with an 89 and then somebody else who does none of that and cheats their way through ends up with a 93, that, to me, is like the definition of being inequitable. Part of the equity of this is who is selected to participate, but there’s also an equitable outcome of does the signal of the grade truly reflect your learning in a way that is fair? Everybody’s grade really is an accurate reflection of their learning, not some people’s grades are accurate reflections, and other people’s grades are super over inflated because they were cheating and we weren’t able to catch them. I think about the equity on those two levels.

Rebecca: I think the transparency that we talked about earlier too, about explaining the process and making sure students understand the process, that plays into equity as well. It’s really important for everybody to understand a system that might be really unfamiliar to them.

David: Not that I’m trying to convince you. I really would want to try to try to convince them that this is not about trying to catch you doing something. It’s a second measure, and if you did well on the first one, you’ll do well on the second one, and you’ll get some additional points for doing well on the second one. I think there’s a way that that can be explained and communicated and practiced that is fair for everybody. If it doesn’t eliminate the incentives to cheat, it can really dramatically reduce it.

John: And one of the things I was just thinking about, as we’ve been having this discussion, is a classic economics paper by George Akerlof called “The Market for Lemons,” which explains why, when you drive a car off a lot, its value drops dramatically. And the reason is, when you buy a new car, you can expect that most new cars are very high quality, but if you look at the market for cars that are a year old, what happens is, we get a very different mix. The high quality used cars are ones people are likely to keep, the lemons that they are having a lot of problems with are the ones that tend to be offered for sale. In the market for used cars, you don’t necessarily know whether this used car would be a lemon or not, and that’s essentially the sort of system we have with grades in the presence of AI. The whole value of a degree and of the grades in classes can very well be lowered if employers and future instructors at other institutions observe that students have not mastered the skills that their grades suggest that they have, and that can devalue education for all the students, including the ones who actually did master the skills we hope that they have. Having more accurate signals is a really useful way of addressing this, and that’s one of the things I really like about this approach.

David: It seems like now, more than any time that I’ve been alive, for sure, that the number of questions about the value of education are already at an all-time high, confidence is at an all-time low, perceived usefulness is at an all-time low. AI just has the potential to make that situation even worse if we don’t figure out some way to really improve the consistency of if you came out of that program with a B average, I know what that means in terms of skills and knowledge and attitudes and things that you’re going to have, but when those signals don’t mean anything, then people stop relying on them. And, I tell you what, you want to see, the bottom really fall out of higher education, we just need employers to stop requiring degrees as a condition of employment. If a degree isn’t the gateway to good employment, who’s going to keep showing up paying what they pay now to participate in this process. We’ve got to do better.

Rebecca: I think the more we share with students the concern that we have about their learning and why we might put measures in place to make sure that their degree does mean something, there’s a lot of motivation there to focus on learning too, having those setup conversations like motivating in a lot of different ways, motivating because you’re explaining why you want to make sure their degree has value, that that demonstrates care, and that’s good relationship building. But also I care that you learn. I think those are things that can be really valuable, and some of that’s baked into the system without explicitly using that language. There’s a piece of care that’s present.

David: I indicated earlier that it’s possible that I’m a softie, but I am one of those people that tells my students on the first day of class that I don’t know you yet, but I’m here because I love you and I care about you, and I think this whole experience can be one that changes your life, improves your life, the life of your family for generations, there’s real reasons that we’re here. I could be making a lot more money somewhere else, but we’re here to do what we’re here to do, and I don’t think you can talk about that enough. I think it is so easy to reduce it to gameplay. I need this many points, and I get that, and what’s the cheapest, fastest, easiest way to get those points? If we don’t call them back to help them think again about why are they here. Tell them why we’re here. Explain what we’re trying to do together. If we don’t do that, we certainly can’t expect them to be responsible for providing all that context and motivation for themselves. That’s part of what we get paid to do.

John: One of the really nice things about your approach is that AI systems are getting better and better and harder to detect. I remember seeing not too long ago a video you had posted of an agentic system going in and completing all the work in a simulated course without any intervention other than providing the login ID, the password, and the URL. And in that environment, we really need to do some deterrence, since we can’t detect it reliably any longer, and this seems like this approach is independent of how AI systems develop, is that an accurate description of your view of this?

David: It is today, and that’s the goal. I don’t know, it’s the academic in me. I expect someone is going to stand up and raise a counter argument at some point that is going to burst my balloon. The way that I look at it now, it really does feel like it steps back from the way we currently think about it, and steps to the side [LAUGHTER] and steps forward again in a way that it really doesn’t matter what AI can do. You could have it log in and do all of your assignments for you fully autonomously. If you get selected for the audit, and we are going to have a conversation about it, and I’m going to watch you do it and talk through it, then it doesn’t matter if AI wrote your essay, or if you paid someone in some other country to write your essay, or if your friend wrote your essay for you, it doesn’t matter how you cheated, because this is just repeated measures of learning. I took one measure and it showed this, and now I’m going to come back and take another sample and try to find out. So I really do believe today that it’s independent of the kind of cheating that happened. Doesn’t matter what tool you used or what approach you take, so long as I can get to you for some secondary, supplemental follow up, however you want to describe it, where we can really be knee to knee, either virtually or physically, and have an experience where there’s not an opportunity for you to cheat in that same way that you cheated the first time. Now is it possible that we could log on and it could be your video avatar that’s being powered by AI. I suppose there’s some scenarios like that, but we’re not there yet.

John: So we should be safe at least for another six months to a year. [LAUGHTER]

Rebecca: So we always wrap up by asking, what’s next?

David: I’m currently playing with some technology to help a person who would want to implement this strategy be able to implement it more easily in their class. I think it’s one thing to hear it described and for it to be in a paper. For example, if it was an LTI tool that you could add to your Canvas, your Blackboard, and it would provide all the support for you and doing the random selection and pulling the subset of questions that students have gotten correct and provided all that support to you, it’d be a lot easier to use these kinds of approaches. I think for me, the next steps on this work are keep talking to people and waiting for somebody smarter than me to show me why it’s a dumb idea. And while I wait for that to happen, try to create some tools to make it possible for people to start using this approach, because we really desperately need some way to combat, some way to slow down, change the incentives around cheating.

Rebecca: Well, thank you so much for joining us. I really appreciated the reframing and getting challenged to think about the problem in a different way.

David: Thank you. This was so fun. What a treat. What a blessing to be able to put an idea out into the world and have other people engage with it and then have a chance to talk about it. This is academia at its best.

John: Well, thank you, and thank you for all the work you’ve done over such a long period of time.

David: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing Assistance provided by Fred Llerena.

[MUSIC]

416. Teaching More Effectively with ChatGPT

The rapid evolution of generative AI tools has introduced an expanding set of educational applications. In this episode, Dan Levy and Angela Perez Albertos join us to discuss how these changes are affecting faculty and classrooms.

Dan is an economist and a senior lecturer in Public Policy at Harvard University where he teaches courses in quantitative methods, policy analysis, and program evaluation. Angela is a graduate of the MPA program in International Development at the Harvard Kennedy School, and is the U.S. Head of Strategy at Innovamat. Dan and Angela are the authors of the first, and now the second, editions of Teaching Effectively with ChatGPT.

Show Notes

Transcript

John: The rapid evolution of generative AI tools has introduced an expanding set of educational applications. In this episode, we discuss how these changes are affecting faculty and classrooms.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Dan Levy and Angela Perez Albertos. Dan is an economist and a senior lecturer in Public Policy at Harvard University where he teaches courses in quantitative methods, policy analysis, and program evaluation. Angela is a graduate of the MPA program in International Development at the Harvard Kennedy School, and is the U.S. Head of Strategy at Innovamat. Dan and Angela are the authors of the first, and now the second, editions of Teaching Effectively with ChatGPT. Welcome back, Dan and Angela.

Angela: Thank you so much for having us.

Dan: Thank you very much.

Rebecca: Today’s teas are:… Dan. Are you drinking tea with us today?

Dan: Yes, Moroccan tea. My family is originally from Morocco, so I tend to drink this one.

Rebecca: Always a good flavor, for sure. How about you, Angela?

Angela: Yeah, this morning I’m drinking chai tea latte. This is my usual morning drink.

Rebecca: How about you, John?

John: I have a tea which actually has a French name, which I’m not going to try to pronounce, but the English translation is chimps’ tea. It’s a black tea with fig and honey and a few other things in it, and it’s really delicious. It was sent to us a while back by one of our listeners in France, which was really nice.

Rebecca: It’s always nice to be able to try new things. I have an Assam tea this morning from Pekoe Tea in Edinburgh.

John: Impressive. She got it there herself, mine came through the mail. [LAUGHTER] We’ve invited you here today to discuss the second edition of Teaching Effectively with ChatGPT. Since some of our listeners may not have heard our earlier discussion about the first edition of this book, can you provide an overview of the purpose of the book and who is the target audience.

Angela: Thank you so much, John. The main purpose of the book is to provide a guide for educators to understand how ChatGPT and other AI tools can be useful for more effective teaching and more effective learning. Now what our main approach is to provide plenty of examples that are practical, to really demystify what AI looks like and how it can look like in and outside the classroom for learning.

Rebecca: In Chapter Two, you describe three guiding pedagogical principles that you use throughout the book, and these principles we certainly talked about on the podcast before and are great advice to anyone involved in course design, whether or not they’re using generative AI. Can you tell us a little bit about these principles and why they’re so important?

Dan: Sure. So the three key pedagogic principles were meant to give the reader of the book an overview of what are the underpinnings of the pedagogic advice of the book. For those of you who listened to this podcast, you might not be surprised in hearing the first one is to be student centered. That sounds very obvious, but yet we spend a lot more time thinking about what we do in the classroom and less time about what our students are doing in the classroom. We spend more time thinking about what we will cover in a course, rather than what our students will uncover. And so being student centered just means paying very much attention, and the focus of the teaching enterprise is stimulating the learning of our students. So that’s kind of that principle. The second principle is to plan for active learning. I think the two of you in this show have said, many times, there’s ample research suggesting that people don’t learn by just listening to someone telling them stories they learn when they actively engage. And the third one is to begin with the end in mind, and this is a little bit of a different way of saying we use backward design. So the idea we start with, what do we want the students to be able to do or master at the end of every learning experience, and then we design backwards from there. And so the book, very much, has these three principles embedded. A lot of the activities that we propose for educators to do have to do with how they can create more active learning in the classroom, and so the three principles embody a lot of what the book is about.

John: Could you each give us an example or two of ways in which Chat GPT can be used to design new classes or redesign existing courses?

Angela: Absolutely. So I can get started with a very cool use that, Dan and I actually did together when we were redesigning one of the classes on the Bayes’ rule in a statistic class, and there was a specific question or activity that was based on COVID and a positive and negative test and the possibility of a false positive and false negative, and we felt that was already an outdated activity by the time that we were working together, and so we wanted to update it to new examples that felt more appropriate for the times. And we could not pin down what could an idea of a different theme or contextualized activity could be. And we asked ChatGPT to just share what could be like 10 potential topics or things that we could explore that would help the students understand how Bayes’ rules can be valuable in a real-world context. And it gave us an infinite amount of ideas that we would have never guessed just by brainstorming him and I together, and just to give like two specific examples that we liked. One was about art forgery and being able to identify fake art pieces, and the other one that we ended up using was about being able to detect AI plagiarism in student work. And so it’s incredibly helpful for anything that has to do with brainstorming contextualized tests and activities to context that are valuable for students.

Dan: Yeah, so similar to Angela, I can think of several examples in terms of designing course, but just to give a recent one, this is more about designing an assessment, similar to what Angela just mentioned. I created a project for a course… and I’m sure we’re going to talk in a little bit about projects… the syllabus has a lot of things, and there was a question that we had created last year that we wanted to change. And so I uploaded the question to ChatGPT, and I said, “Can you tell me what you perceive to be the learning goals of this question that I just gave you?” And I think it did an amazing job, better than I could have stated them myself. This was very helpful. I think part of it is because it had access to the goals of the course, but I think it was very helpful in helping me be better at interacting with ChatGPT in the refinement of this learning activity. So that’s just like one example. I realized both examples that Angela and I gave are in the realm of assessment, but it turns out to be a very important realm in terms of designing sort of a new session or new course, the book goes over to what I think, still today, is my favorite and maybe Angela’s favorite activity. This was kind of maybe the spark of the book. We were trying to think about an activity surrounding the Cuban Missile Crisis. And over two weeks, basically had a conversation with ChatGPT along the lines of, “help me design a better class session on this topic that I don’t know too much about.” After I saw that, I thought this has a potential to be an incredibly useful tool for teachers and learners, as Angela said.

Rebecca: Your first example seems like a really good reminder that we can use tools like this to help implement things like TILT (or transparency in learning and teaching), having it help us be more clear about what the course objectives or assignment objectives are, so that we can then convey that information to students.

Dan: Right. And you might have the reaction, but how can you not know? It’s not that you don’t know, it’s that maybe some of them are subconscious to you. I mean, one of the most useful things that a teacher can have is someone, or in this case, it’s not a person, but put a mirror to what you’re trying to do in the classroom. So that’s why having a colleague come and see you teach is so valuable, because you might do things and you might not realize why you’re doing them, and maybe having someone ask you or sort of give you ideas for why you’re doing them, it’s helpful. The book is based on the premise that AI can be an incredible assistant, but also an incredible thought partner. And I think both of those examples have those two roles being played quite intensively.

Rebecca: One of the other things that you certainly have examples of in the book is the way that ChatGPT can be used during class sessions. Can you give some examples of this?

Dan: Yes, we can give several examples. So, one of them, it’s an example that is described in the book. But our colleague, Mitch Weiss, who teaches at the business school, has a wonderful activity that is about discovering the capabilities of AI, and students use that in the classroom in a case called Storrowed. And so probably not the best use of time here to describe the full activity, but suffice it to say that students are put into two groups, and they’re kind of competing to be able to see who comes up with the best ideas, but they’re able to use AI for proposing some of their ideas. So that seemed like a very interesting one. The book also highlights two examples of using AI to summarize student input in the classroom, and I think that’s an incredibly powerful tool. I think a lot of educators have heard about the one minute paper, the way that it typically worked, at least until not too long ago, was you ask people to give you what was the main idea that they learned, or maybe the muddiest point, or whatever it is, at the end of class. This is incredibly valuable, but normally you would just get out of the classroom and then have to read and sort of think about how it works. And now you can do it in the classroom, live and understand what are the main themes that people are picking up. You can also do it with questions. And then a very exciting use that I experimented with a month ago is with group work. I don’t know how much you have been in situations where you assign students to work in groups, and then 10 minutes later, you’re trying to debrief. But debriefing is a little bit hard because you don’t really know what happened in the groups. You might circulate, but if you circulate, typically, the moment you approach a student group, the conversation changes, so you’re not really listening to what they’re describing. And so what I did is to build a simple custom GPT that did the following. So the students were put into groups, and they were asked to input their answers to the activity in a Google slide. So this was typical use, especially during COVID. You did breakout rooms, everyone fills in a slide. As soon as they’re done doing that, what custom GPT does is it produces four slides for the instructor. So this is in a matter of a minute. And so the first slide has the main themes. So of all the groups you had in the classroom here, the main themes that were emerging. The second slide has particularly novel or interesting ideas that emerge. But not only they tell you the idea, but it also tells you it was group three that came up with this idea. So you can use that in the debrief, “Hey, group three, I noticed you came up with this idea. Can you expand on this?” The third slide tells you about points of contrast or tension between what the groups did, and then the fourth slide gives you some ideas for how you could debrief. And so the fact that all of this is available within one minute of students finishing the group work, I found it to be incredibly empowering to lead a much better debrief discussion. So those are the kinds of uses that I think advance learning in the classroom in a way that it would be very hard to do without a technology like this.

John: You have a chapter discussing how ChatGPT can be used to either design or grade assessments, and there’s been a lot of discussion on this in various social media forums and also in some editorials and articles and so forth, in terms of whether generative AI should be used to create assessments and to grade student work, with some students being concerned that they’re not getting the direct feedback from their professors. Could you talk a little bit about what ways might be appropriate and how this may be helpful.

Dan: So, I would like to separate the question into two parts. One is about designing assessment, and the other one is about giving feedback in terms of assessment. I am convinced that for designing assessment, AI can be incredibly helpful without sacrificing the voice of the instructor or the learning objectives. And the two examples that we gave at the beginning, I think, are suggestive of how can they be used. It can convert you in a much more creative teacher, because you have a thought partner with which to bounce ideas. It can give you feedback on the assessment that you’re doing. My colleague from the business school, Mitch Weiss, again, he has this custom GPT that is like eliminate ambiguity in my exam questions. And so he gives the exam question to the AI tool and essentially “write it in this way so that there’s less ambiguity for students.” So you could imagine ways in which the design of assessments can be improved with the help of AI. I’m not saying we should farm out the design. It’s like, “okay, I’m teaching this class, and now you go, AI, create the assessment, and that’s it.” But I do think, in my mind, whatever drawbacks there are, weighing everything, I think the impact is positive. For providing feedback on assessment, I think you have decided this is a big topic, a lot of controversy about student expectations on assessment and so on. I am not going to tell you that I’m convinced about what the right answer is, but I would say, as we describe in the book, there’’re both benefits and risks associated with doing this. But what I would like to suggest is that what we know from the science of learning is that students benefit from both practice and timely feedback on that practice. So that we know. And so the question to me is, are our students getting enough opportunities to practice and timely feedback? And my sense is the majority of us educators feel that the answer to that question is, no matter how hard we work, it’s very hard to provide timely feedback to students of the sort and in the quantity that they need to really, really make a difference in terms of their skill development. And so what we end up is with assessment strategies that are the most feasible given all the constraints that we face. But I think we’re leaving some value on the table in terms of our students being able to practice. So I’m not suggesting that AI should do all the assessment work for us, but if it could work as a complement to the work that we already do, I think this could potentially be a useful thing. Just to be super concrete, maybe if you had five ways of assessing students in the semester, but you’re only able to grade two because of time constraints or whatever it is, maybe you can use, in a very transparent way, here’s AI to help you provide feedback on the assessment. Now, of course, this has to be done with a lot of care and thought, but we can’t do it by hiding from the students that we’re doing it. And also, I think we need to explain to the students why we’re doing it, and if we’re just saying to ChatGPT, “just give them feedback without our input,” then our students, very justifiably will say, “Well, what are you doing? So I can do that.” So I think there is thought that needs to go into how AI is used to be able to do it. Just to close, we have here the Kennedy School, a course taught by Shard Goel, Teddy Svoronos, and myself, and we have our Teaching Fellow, Calvin Isley. He basically helped us design a system by which the assignments would have a preliminary assessment done by AI, and then our teaching team would take into account that and use it as input for grading the assignments. Students were able to opt in to this pilot. And what I think happened is the students who opted in got more detailed feedback than they would have gotten in a system where we’re just counting on human labor. So, in sum, I’m not saying AI should do everything in providing feedback, but I do think there’s a role.

John: And humans get tired. And if we’re providing feedback on many students, the quality of that feedback may very much vary over time in a way that it wouldn’t if you’re working with an AI tool for assistance and at least providing a starting point.

Dan: For sure. I mean, one of the things we say in the book is that AI has like, two very important virtues: it’s available 24/7, and it has infinite patience, and I don’t know many human beings for whom we can say that.

Rebecca: One of the things that you just highlighted was student practice. And you have a number of chapters in your book about how ChatGPT can help students learn. Can you give us some examples in which ChatGPT can support student learning?

Angela: Absolutely, these were one of the chapters that I was more excited about writing, because when we find many of the books that are there is that they focus a lot on the teaching side of things. But there’s so much that as educators, we can do to just teach students to use ChatGPT in a way that is beneficial for learning. And just to give a couple of examples, the first one I’ll give is in continuation with this idea of practice, we have an example in the book of one student that was studying medicine, and she was learning and preparing for an exam, and as she was finalizing her preparation, she wanted more activities to practice, because she had already completed all of the activities that her teacher provided. And so she just asked ChatGPT, “Can you create more questions? These are some of the questions that I received. Can you generate more questions and then don’t give me the answer? I will give you the answer, and then you will let me know if it’s correct or not.” So it’s just an extension of this practice that, again, it’s one of the things that for educators, it’s hard to provide enough practice opportunities for students because it means generating more questions and more questions that cannot be used for formal assessments. So that’s one possible example. The other possible example, it’s the personalization of learning, and this, again, it’s one of the big moments that for Dan and I, it just sparked our interest even more for what AI could do for learning. And the specific moment was related to time when in one of Dan’s classes, he encouraged his students to learn about risk management strategies before the class itself, and then he asked. The students to share what the conversation with ChatGPT had been like. In addition to that, he nudges students to ask ChatGPT about risk management in a certain way. So for example, sharing what context they had about risk management strategies, etc. And we basically saw very different conversations depending on the student. You had, students that were much more familiar with risk management strategies and the kind of explanations that they received were much more technical, much more detailed, less intuitive for somebody that would be just exposed to this concept. Well, for some students, it was the complete opposite. It was a much more intuitive, just like high level conceptual explanation with lots of examples that could be relatable. And this is something that would be very hard for an educator to replicate, a specific one-on-one explanation that would fit the needs of every single student. So many multiple uses for students that can really help improve learning.

John: In the spring of 2025 we had a reading group on campus that read your book. And just as a little bit of an aside, one of the things that happened is, right at the very beginning, people suggested that people would experiment with something, try it, and then bring it back and report at the next reading group meeting. And that worked really well. It’s a great book for a reading group. But one of the things that surprised people a little bit who had experimented with ChatGPT was the ability to customize it to meet your individual needs. Now some people weren’t able to do that very well if they were using it in a wide variety of applications, but now, with the new edition, you’re able to talk about projects in ChatGPT. Could you talk a little bit about how you can customize ChatGPT in general, and also how you can use projects for a similar sort of purpose?

Angela: Absolutely. So to put it simple, because sometimes I feel it can be hard to get lost in all of these, like features and new releases of ChatGPT. The idea here is that you can customize ChatGPT so that it’s more responsive to your needs or your specific context. And to put this into an example, my specific context, for example, may be I teach an economics course to first grade university students. But the reality is that I don’t use ChatGPT only for my classes. I may use ChatGPT also for other uses. And so what openAI has rolled out in this past year is the possibility to customize the responses of ChatGPT at different levels. The first level is to have instructions that apply to any chat that you open in ChatGPT, and that’s what system instructions do in chat GPT. For example, you can keep this to a very general level, like, “I’m a woman, I’m 30 years old. I do this as a profession. I’m from this country, etc, etc” things that may be relevant. For cases where you may use ChatGPT frequently, but it doesn’t apply to any use of ChatGPT that you do, you may want to choose projects instead, where you may have a project for a specific course that you teach. You may have a project for something that you do frequently with ChatGPT. Perhaps you were talking about travel planning before, like, perhaps you travel plan a lot, and you want to have a project specifically where you can provide instructions around the kind of travel that you like to do and the kind of places that you want to go to and the kind of things that you want to do when you go on traveling, and so that’s what projects allows you to do. It allows you to have instructions that you don’t need to be repeating every time you open up a new chat, but then don’t apply to all of your chats within ChatGPT.

Dan: The only thing I would add to what Angela just said is that not only can you add instructions which are very helpful in customizing, but you can add files, and those files can be incredibly helpful. So, for example, for each of my courses right now, I have not only instructions about teaching philosophy, goals of the course and so on, but also the syllabus and relevant material that I think has helped, as Angela said, the advice that you get from ChatGPT be much more customizable. Back in the old days, and old days here means 2023, when you were writing one chat and then you would begin another chat, it was like two completely independent conversations, and now I think AI is much better at understanding through not just the customizations that we just talked about now, but through the use of memory and so on, it’s much better understanding who you are as a user. I think one aspect of the progress in LLMs, ChatGPT in particular, but I think it’s true for all of them that we have picked up in the second edition of the book, is increased personalization and customization that is now available to a user to be able to get more customized and specific advice from their LLM. And the last thing I want to say is that everything that Angela said is very, very, very, very easy to do. You don’t need to code. You don’t need anything. If you know how to prompt ChatGPT, you can create a project, and it’s as simple as you put in instructions, you put in files, and then you create the chats within that project, that’s all you need to do.

Rebecca: You’ve hinted at some ways of taking advantage of projects. Can you give one or two specific examples of where this could benefit student learning?

Dan: So I think the easiest way Angela mentioned is you can create a project for each of your courses. Once you do that, then a lot of the help that you asked ChatGPT would be useful for creating activities for students, helping you design assessments and so on. So that’s one use of projects. If you want to create something that students consume directly, then you would move from projects to custom GPTs. Custom GPTs can be shared with students and they can interact with it. In the first iteration of custom GPTs that were most popular were the tutor bots, so where you might ask students to interact with a tutor bot, for example, to learn some skills in whatever subject that it is that you teach, but where the instructor was playing kind of a design role, was adding materials and so on. I think those tutor bots continue to be helpful. One of the things I think, I suspect many of us have discovered is that if the tutor bot is not part of the natural workflow of students, then it becomes a little bit harder for them to benefit from it. So if students are using ChatGPT as their AI tool or any other AI tool, it’s going to be hard to move them to the custom GPT or to the tutor bot, unless you sort of have very directed task at them doing that. But there are many, many other ways in which you can use custom GPTs with students. And before we were talking about giving students an opportunity to practice, but imagine you were teaching negotiation, for example, you could imagine, and there are some available out there, bots that allow students to practice negotiations or public presentations or so many other skills. So if you teach any skill that you think benefits from repeated practice and feedback on that practice, I think the potential of AI to help you and your students in that task is immense, and custom GPTs are a way to make this happen. And again, the book is called Teaching Effectively with ChatGPT, so we put our stake in the ground with selecting one LLM, partly because it was most popular sort of tool. But if you’re using some other LLM, like Claude or Gemini, the equivalence of everything we have talked about exists. So Claude has projects. You can build custom GPTs in Gemini, through what they’re calling GEMs and so on. So I think a lot of these companies are all building and competing with each other in this space.

John: And in addition to providing those tutor bots, you can also design them to help you with course design, things that will generate things in the TILT framework, for example, or make suggestions for adding more active learning activities. Pretty much any of the things that you talk about in your book, if it’s going to be done repeatedly, could be turned into a custom GPT. And your book was really good at providing them, that was a really popular chapter with the reading group.

Dan: Thank you. I just want to say one thing. So since the first edition of the book, projects became a thing, and I would say, I think building custom GPTs is relatively easy, but if you’re only going to use that tool for yourself, then building projects is an even easier way to accomplish the goal. Obviously, custom GPTs have the advantage that they’re shareable, so you can share not just with students, but with colleagues and so on. But if you’re getting started and are feeling somewhat intimidated with how do I do all this stuff, projects is a very, very natural step up from your ad hoc use of ChatGPT.

Rebecca: There’s still a lot of faculty who maybe would prefer to ban student use of AI or focus their efforts on trying to detect student AI use. Are there any ways of reliably detecting AI use?

Dan: So I’m not a computer scientist, but the ones I speak with give me pause that we will ever be able to really do this at a level of reliability that suggests that we can take action on this. My general sense is that this is a lost battle in the battle of an instructor against students. I think students always win, is my sense. If this is the battle that we’re fielding, we’re not going to win. I think we have to do differently. But just to give you a sense of for why I think it’s so hard, is that now AI is so embedded in everything that we do, that I think it’s going to be very hard. Back in the old days, again, 2023, you could ask students, okay, go to ChatGPT and then have a conversation and then give me the transcript. But now it’s embedded in the things that we do. It’s auto complete things in Word and so on. And more importantly is that you could, as a student or as a human being, produce an essay by yourself. You created all these ideas, and then you give it to ChatGPT and say, “Please improve the clarity of my writing.” That’s all you’re asking it. It will add a few em-dashes and a few words, and then everyone will say, “this was written by ChatGPT.” And you might decide, as an instructor, particularly if you teach writing, that even the use that I just described should not be allowed. But I’m going to suggest that at least for some subjects, that use is fine. And if that use is fine, an AI detector is going to say that this was created by AI. I think we’re in trouble.

John: Since the first edition of your book, openAI has released ChatGPT versions 4 and 5 more recently. You mentioned projects, but what are some of the other changes that have been added to ChatGPT since then?

Angela: Great question. So projects, as you very well mentioned, is one of the main features that was released. There’s another very interesting one, which has to do with the models that are available within ChatGPT. So before ChatGPT 5, there were multiple models available, as most people in the audience would know, where there was a basic default model, which was valuable for most of the tasks. And then there was a deep research model that is valuable for in-depth, multi-step, more complex tasks that require time to think and process before being able to generate an output. So this can be useful, for example, to summarize academic research, to create classes or provide feedback on longer documents, anything that is more complex than the average task. Now, what used to happen is that experienced users knew about this model, and they would change the model that they wanted depending on the task that they were wanting to do, but non-experienced users would just default to the basic model and would not be using the most out of the capabilities of ChatGPT and so with the release of ChatGPT 5, one of the key developments was that ChatGPT 5 automatically selects the model that best fits the task at hand. And so this is a great improvement in terms of being able to really put at the service of users all of the capabilities that ChatGPT has. One of the important things is that when it was first released, there was a big controversy, because some people claimed or complained about the fact that ChatGPT was not able to always optimize for the best model, and they wanted to retain that agency. And since then, openAI has backtracked and still provides everybody the ability to be able to choose the specific model that they want to use.

Dan: We’ve already alluded to this, but to me, a big development in the last year or so is the ability to customize and personalize your experience, not just through projects, through memory. and other things that we go into in the book. In fact, the chapter that we used to call Creating Custom GPTs, we’ve now broadened the title of that chapter to include many other ways in which you could customize and personalize your experience and that of your students through the use of this feature. So that, to me, feels like obviously ChatGPT 5 is better than ChatGPT 4, but back then, now there are these reasoning models that are automatically selected and so on, is useful. But the thing that you will notice more as a user, I think, is the potential for customization. And obviously the reasoning models can be very, very helpful for more complex tasks, deep research, which Angela briefly mentioned, is another important advancement for much more complex tasks.

Rebecca: As we’ve noted in this conversation and also in your book, generative AI platforms change quickly. [LAUGHTER] And when we talked to you in the first edition, you mentioned having a companion website that you would continue to update. Are these updates something that are going to continue with this edition?

Angela: Absolutely, that’s one of the main advantages of the companion side, the ability to be able to update it. As we know this is a fast evolving space, and that we felt that was a good format to be able to digest information. But one of the main inconveniences is that it’s not life, and so users and readers and listeners can always go back to the companion side for more up-to-date information.

Dan: And the companion side has all the prompts that were used in the book, including, for those educators that shared with us their prompts behind their custom GPTs, they’re also there. The site is freely available, and then it also has a lot of the examples from educators.

Rebecca: And we’ll make sure we have that link in our show notes.

John: And people found it really valuable when we’re doing the reading group to have all those examples there so that they could modify them for their own needs. We always end by asking: “What’s next?”

Dan: Well, I want to say something that’s next that is, thanks to Angela. So I think we are maybe one or two weeks away. Hopefully by the time you publish this show, it will be out. We have a edition of the book in Spanish that Angela gets all the credit for making it happen. So we hope that means that more readers around the world will be able to use some of the insights from the book. If you’re a listener and want the book to be available in some other language, let us know, and we’ll try to see if we can maybe collaborate with you and make it happen.

John: And Angela, what’s next for you?

Angela: Getting on a plane, flying back home for the weekend, and getting that book ready.

John: Well, thank you. It’s been great talking to both of you.

Angela: Thank you so much for having us. It’s always a pleasure to come and have tea with both of you and to talk a bit about AI developments in teaching and learning. So always a pleasure. And thank you so much. Hopefully we’ll see you again sometime soon.

Rebecca: Safe travels.

Angela: Thank you so much.

Dan: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing Assistance provided by Madison Lee.

[MUSIC]

413. Faculty Perspectives on AI

Faculty adoption and use of AI in higher education varies widely. In this episode, three colleagues from the University of Mississippi: Josh Eyler, Emily Pitts Donahoe, and Marc Watkins, provide their perspectives on AI use in higher education. Josh is the Senior Director of Center for Excellence in Teaching and Learning and Assistant Professor of Teacher Education, Emily is the Associate Director of Instructional Support in the Center for Excellence in Teaching and Learning and Lecturer of Writing and Rhetoric, and Marc is a Lecturer in Composition and Rhetoric and Assistant Director of Academic Innovation.

Show Notes

Transcript

John: Faculty adoption and use of AI in higher education varies widely. In this episode three colleagues at one institution explore different perspectives on AI.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Josh Eyler, Emily Pitts Donahoe, and Marc Watkins. Josh is the Senior Director of Center for Excellence in Teaching and Learning and Assistant Professor of Teacher Education, Emily is the Associate Director of Instructional Support in the Center for Excellence in Teaching and Learning and Lecturer of Writing and Rhetoric, and Marc is a Lecturer in Composition and Rhetoric and Assistant Director of Academic Innovation. Welcome back, Josh, Emily, and Marc.

Marc: Thank you. Thank you for having us.

Emily: Thank you.

Josh: Thanks, great to be here.

Rebecca: Today’s teas are:… Josh, are you drinking tea?

Josh: Well, you know that I never drink tea, but I have a delightful water here.

Rebecca: I will continue to say that that’s the foundation of tea.

Josh: It is.

Rebecca: How about you, Emily?

Emily: I’m appalled that so many people come on this podcast and don’t drink tea. I am drinking a Scottish morn tea, one of my favorites.

Rebecca: Very good choice. It aligns with mine this morning. And how about you, Marc?

Marc: I am drinking a tea called Red Dragon that my wife got for me in Portland, and it’s quite good. I really recommend it. It came from Willow Tea House.

Emily: That’s what I like to hear.

Rebecca: Very nice. That sounds like legit tea.

John: And I am drinking a black raspberry green tea.

Rebecca: That’s a good choice. There’s no cherries involved.

John: I almost did.

Rebecca: Oh yeah, I know. And I have a Scottish morning tea.

John: We’ve invited you here today to discuss the impact of generative AI on teaching and learning in higher ed. We see a pretty broad spectrum of opinion from faculty. Some faculty oppose all student use of AI because of concerns it will impede student skill development. Other faculty have fully embraced student use of AI and argue that we need to prepare students for their futures in a world in which generative AI tools are ubiquitous. Most faculty, though, are somewhere in between those extremes. To start this discussion, could each of you provide an overview of your general position on AI use in higher ed by students?

Marc: Oh, boy, that’s an interesting conversation to begin with. I don’t really know about my own personal stance, if that’s even something that I have much control over, because so much of this isn’t being directed by faculty members, it’s being directed by students bringing these products into the actual conversation. I think that changed a little bit last year when the new dimension came where we had a lot of this technology starting to be embedded in existing university systems. So I try to provide my students guidance. I don’t advocate for them to use AI in assignments. I don’t make them use AI in assignments, but I do make it clear that if they are going to use them, they have certain responsibilities, and they also should have certain expectations from me as a teacher to be a little bit more informed about this, and also from their peers, if their peers are using AI with them in some way, shape or form, too. So really, it’s about just kind of guiding them in certain ways. What this is going on depending on what style class I teach. If it’s a small low-enrollment class too, I’ll have a values-based statement when I talk about… I invite them to discuss how they’ll use or not use AI, versus a large class that might be a writing intensive course too, like our first year writing classes, where I use a stop light approach, where Emily has actually helped develop this too, about red light, yellow light, and green lights for students to do so.

Josh: I thought a lot about what I was going to say today, because I want to not seem like I’m the physical manifestation of that Simpsons’ meme of “old man shakes fist at clouds.” So what I’d like to lead with is that, as a faculty member who believes very strongly in shared governance and academic freedom, I completely support any of my colleagues who want to experiment with it, build it into their courses, support students, all the things that we’ll hear about in some shape or form today, but as an individual faculty member for my own courses, I will never build intentional use, and I want to clarify what that means in a second, intentional use of AI into my courses, and I don’t want students to be using it either. And what I mean by intentional, because I know Marc very rightly says people are using AI even if they’re not aware that they’re using AI, right? If I’m using Google and I forget to phrase it correctly, AI is giving me things like that. So when I say intentional use of AI, what I mean is me as the instructor intentionally building it into courses or assignments or activities, and students intentionally going to AI to do the work of the course or engage cognitively with it. So there are labels for these things. I don’t put a label on myself, “refuser,” “resistor,” things like that, but that is my own position pedagogically.

Emily: Yeah, I think I agree with a lot of what Josh and Marc have already said. I think on a curricular level, students should be exposed to generative AI at some point during their college careers, because they’re going to encounter it as students and kind of beyond school in their professional and personal lives. So I think it’s important for them to know what it is, how it works, and to develop I think what some people have called critical AI literacy, understanding the kind of implications ethically of AI use,. understanding who benefits from AI and who benefits from the narratives that we have around AI. And so I think all of that is important. There are some classes where it’s really appropriate to incorporate AI and think about how it might be used in a specific field, and some classes where I think it’s totally inappropriate. In my own classes, we do talk about AI a lot, and I give students the choice of whether or not they want to engage with it. And I think it’s interesting that Josh brought up the word intentional, because I was going to say what I really want for my students is for them to use AI intentionally if they use it, or to refuse to use AI intentionally, if they want to refuse it. And by that, I mean they should know kind of where it is appearing in their lives. And they should be able to say, “I want to use AI and I want to use it for this specific purpose, and here’s why,” or “I don’t want to use AI for any of these purposes, and here’s why.” So one of the things that I think I want to work on most in my own classes is developing that sense of intentional use, instead of just like, “uh, I guess I’ll use it because I’m late on this assignment, or I guess I’ll use it because everybody else is using it. Or I guess I’ll use it in this way just ‘cause.”

Rebecca: I think all of you have kind of mentioned this idea of intentionality around student use. Can you talk a little bit more about why the frame of intentionality… I mean, I think we can kind of make some guesses… in relationship to maybe some of your general concerns of students using AI.

Marc: Sure. I think it’s deeply important that everyone in the process is intentional about AI, whether you’re a teacher bringing it up in the class, setting up a course policy too, or thinking about designing certain assignments where students might be navigating the technology or might be using tools to bring in assessments that you would think you don’t want them using it for too. So you have to look at this from the basic framework of every single person that’s going to be involved in that sort of process: student, teacher, in some cases the actual librarians were actually helping them navigate resource materials that are available to them, maybe even administration. We want everyone within this process to be intentional about the technology. Just the past month alone, we’ve had actual reports from police officers using it to write a crime report, lawyers using for criminal and civil cases, all undisclosed. We even had a report of a judge using it. We want to really be thoughtful about how this technology is playing out in broader society, too, and start thinking about our roles as educators, and how we can start talking with students about what is ethical, what’s appropriate, and what is actually a good, helpful use case of this tool. I’m very concerned that we’ve been looking at usage in terms of augmentation, collaboration, automation. I think we can handle the first two in some ways. They all have their own ethical issues with that, too. But I’m really getting more and more concerned with the automation aspect. We are starting to see students also do the entire assignments. We’re also seeing these new tools, like AI agents with promises of being able to completely automate the entire coursework for students and also for faculty too, and actually grade and assess student writing and other types of student assessments.

Josh: Yeah. I mean, for me, intentionality means a couple of things, but I think awareness is an important pillar of intentionality, knowing all of the different questions that one must consider before engaging in intentional use of AI. So what are the ethical dilemmas? What are some of the key questions? And for myself, the ethics are divided into two categories. There’s the outside of the classroom ethical debate and the inside the classroom ethical dilemmas. So how are people aware of the choices that they could potentially make, and then a critical thinking process for working through the actual decision to use AI.

Emily: I think for me, the reason that I’m concerned a lot about intentionality is because I’m really concerned about student agency. And my biggest concern about AI is the way that it can take away our agency. Agency is something I really highly value in my teaching… I think, a guiding principle for my classes, as well as one of my most desired outcomes for students. I want students to leave my class with more agency in the world and the skills and knowledge to make better, more informed decisions for their lives. And so I have a lot of concerns about how AI, but also just our general technological developments in this moment are making it really easy for people to give up their agency. So I’m thinking of things like in the book filter world, Kyle Chayka, and I’m not sure I’m pronouncing that name correctly, but they talk about how algorithms control our tastes and our consumption and our lives. It kind of used to be the case that we would go out and seek out the media that we wanted to consume, and now a lot of that is kind of determined for us, and so there’s this, I think, huge loss of agency. And you could say the same about generative AI. When you allow AI to tell you what something means, or what you think about something, or what to say about something, you’re giving up some of your agency, and you know a little bit less about what you think, or what you want to say, because you haven’t really kind of wrestled with that. So I think giving up your agency in that way can be really seductive for students who are under a lot of pressures and when the tasks that we’re asking them to do are very difficult cognitively, but it’s also really important and rewarding to go through those processes and develop that sense of agency, and I don’t want students to give that up just because they think they need an A in the class, or because they don’t have the time to finish an assignment. So intentionality is so important for me because it’s a huge part of agency.

John: So there’s a lot of concerns that you’ve all expressed about students using AI to reduce their agency and to develop their skills, but are there some ways in which student use of AI might perhaps support student learning, or might also reduce some of the equity gaps that we observe because our students come in with very different backgrounds, and it’s hard to design courses that will address all the needs of all the students in classes.

Marc: Yeah, I think we’re seeing some really interesting experimental research designs coming out of California, but they have something they’re called PAIRR, which is peer review from a human being, followed by AI, followed by review by instructor, and then actually the reflection and review all put together. So it’s an augmentation type of process in that way too, where you get a standardized peer review and a writing sort of session too, and you can add AI on top of it. But what they’re finding is that the students that use the AI can actually go through multiple rounds of peer review, then the actual teacher can review both the peer’s comments, the AI’s interactions with them, and ask the student to reflect on the entire process holistically. So it’s able to do a lot more in terms of giving feedback and rounds of feedback than it is just from one human being or a series of human beings. We’re also seeing lots of other different examples come out here about AI being used in research and actually looking to use different types of breakthroughs, but also being used to support students and different student learning outcomes by going through a process of coming up and attaching different research points to student learning processes throughout this. Sometimes this can be as simple as a static sort of like summary that you give to students. Sometimes this could be as wild as an AI voice, which I have some problems with, talking with the student too, and interacting with them. So from every sort of response, there’s a lot of caveats about what we’re seeing from the research studies and databases popping up here with AI in them. But it does look like that if it is intentionally designed, there are some benefits for students and student learning.

Emily: I feel like I should jump in here to build on what Marc is saying, because I suspect that Josh is going to have a totally different take. So in my view, I think that the question about, are there ways that students can use AI to increase learning is really less about the kind of particular usages and more about student attitudes toward AI, or the kind of orientation they take to using AI. So I’ll give an example to illustrate kind of what I mean. A couple of years ago, I surveyed students at the end of this semester in my writing class about their uses of AI, and I had one student say I used AI to correct my grammar, and so I learned a lot from that. And then I had another student say I only used AI to correct my grammar, so I didn’t learn very much. And so the difference is not about the usage. They were both using AI in the same ways, but it was about how they were approaching that use and what they were trying to get out of it. One student was approaching it with the idea that AI could help them learn something here, and the other one was approaching it with the orientation of AI can kind of correct a product that I’ve created. So I think that the key is that when students approach AI, they need to be approaching it with a learning orientation, or they’re not gonna get much out of it. I mean, this is just like anything. And I think the students you interviewed on this podcast recently, Kaija, was a great example of that, and one of the things that she said is that she really wants to figure out things for herself, and she uses AI in ways that support her in figuring things out for herself. I think the trick here is that students don’t always know where the line is, and have trouble determining when am I actually learning, because we don’t often ask them to think about that. So I think one of the things that we need to be doing right now is working with students to develop their metacognitive abilities. If they’re going to use AI, they need to know what’s going on in their brain when they’re using it, [LAUGHTER] what specifically they’re getting out of it. So, what uses of AI can be productive? I think lots of them as long as students are able to understand and articulate what they are getting from their use of AI. I think the question about equity gaps is maybe a little different, and I feel like I need to think more about this, but I’m a little suspicious, to be honest, of a lot of discussion about using AI to reduce equity gaps, because I have a kind of suspicion that AI might just Be a band aid here on a larger problem, and we should think about how to fix the larger problem, like if students are not educationally prepared, or if they are going into a workforce situation where they’re being judged unfairly by, for example, their proficiency in English, then I think we need to change the thing that creates the equity gap in the first place, rather than trying to paper over that with AI. I’m not saying that AI can’t be used productively to reduce equity gaps. But I’m a little suspicious of some of the ways that have been proposed for that.

Josh: Right, that’s a really great point. So I’m deeply skeptical that either of those two things that you said, John, will ever be true. I think the major issue right now is that, as Marc was indicating, the research is so nascent, it’s just developing. So for every study that someone holds up in support of AI and learning, I can find one that says the opposite. We’re just at the early stage where it’s more like a tit-for-tat kind of process than it is revealing anything that I think we can draw on to make truly evidence-based claims about learning. So I think at this moment, we are embedded in true academic argumentation. We’re bringing to bear research that we know from other fields and other areas and other aspects of the cognitive processes of learning, and applying it to the AI question right now. Now that will change over the next few years. We’ll get more research. We’ll be able to, I think, draw together claims that rest on a more solid foundation, but I would need someone to be showing me evidence that using AI is more effective for learning than other strategies that we have. It’s really as simple as that. I think Emily has a great point about the equity gaps here. So I think we learned a lot during the pandemic about access equity, specifically with technology. In Mississippi, especially, we were learning a lot about what access students had to technology and how that was unequally distributed. And so that’s one aspect of it. Another aspect is, when we talk about equity gaps, we’re talking about systemic problems and systemic solutions. So what systemic problem is AI supposed to be addressing? And how is it a systemic solution? And so I think that it’s hard for me to see how AI is a better answer, or a more productive answer than other things that we have been talking about and looking at that themselves have been incomplete answers to the equity gap problem, but that are drawn from a wider body of research at this moment.

Emily: Just to respond to what you said, Josh, you’re making me think that in the same way that I was arguing AI can paper over or be a band aid for pre- existing equity gaps. I think it can also be a band aid for problems in the learning space, in that a lot of people are turning to student uses of AI, or even instructor uses of AI, because it has become unmanageable to teach so many students at once. for example. Students need AI feedback because the instructor can’t provide the volume of feedback that’s needed. And so, I mean, I do think that insofar as that’s happening, we should be looking at what are the problems in the learning space that we can and should fix. Can we get smaller class sizes? [LAUGHTER] Can we get more support and time for instructors in that kind of thing, instead of turning to AI? And we see a lot of universities making these huge investments in AI tools or access to certain kinds of AI platforms, when they could be using that money to support the kinds of learning interventions and teaching support et cetera that we already know can improve student learning. So I do take your point about that, and I agree that there are a lot of problems. [LAUGHTER] I think ideally AI can be used by students in learning focused ways. But the trick is that it’s actually hard to do in practice.

John: In terms of equity gaps and fixing the underlying problems, that ultimately is what I think we should be doing. But those problems start back in our elementary school system and go all the way through with inequitable support and funding and quality of education, and we do have students who are the product of that system, and we do want to do the best that we can to address the students we have. And one of the things we do know is an effective way of doing this is small classes, one-on-one interaction, and providing more tutoring support and so forth. But we also know that first-gen students and students who perhaps face some degree of imposter syndrome are less likely to be going in for extra help, less likely to visit office hours. And if we could break that down, then we may not need as much support. But one nice thing about having Socratic tutor AI systems, potentially, is that they’re available 24 hours a day, seven days a week, and they’re available to everyone without any fear of being embarrassed by admitting to someone that you don’t understand something. My personal view is, I think there may be a role for some of those things.

Rebecca: John’s in the debate now.

John: Well, I just built a Socratic tutor [LAUGHTER] for my class of 270 students.

Josh: So here I am John, shaking my fist at the cloud, because when I hear you say that, my mind immediately goes to Justin Shaffer’s high structure course design. He was on campus presenting a lot of different workshops, which is a pedagogical strategy to handle exactly what you were just talking about in a way that doesn’t involve AI. I mean, there are analogs to this question, Emily and I think a lot about equity gaps related to alternative grading practices. And as soon as you begin to peel one layer of the onion, what you see is that every aspect of our educational systems in America are wrapped up in the question of the problem of grades, and I won’t go on that train, we’ve been there before, but I’m saying that there are analogs to the question of AI and equity gaps, and that Emily’s metaphor here of band aids is, I think, really an instructive one for thinking through the ability of AI to address those gaps.

Rebecca: We’ll put a link to that podcast that discusses those learning assessments in the show notes.

John: As well as a discussion of Justin Shaffer’s work. And I do use some very similar techniques, but even then, students, when they’re working outside of class, sometimes get stuck on things. And it might be nice to have some sort of tutor, if it’s designed in a way that doesn’t directly just provide answers, and that’s hard to do, because most systems do break down when they’re repetitively queried.

Emily: And I’ll add, to carry the metaphor of band aids, sometimes a band aid is all you got, right? And if a band aid is what you have, like, systemically, we need something else; on an individual level, if what you have as a band aid, and the band aid is helping like, I don’t judge anybody for that, right? We all have to use band aids sometimes.

Marc: Yeah, I think that we should be thoughtful about this in terms of not just students and interacting with them, but John did mention K through 12 and how this is actually becoming a major issue within their landscape. In fact, the Walton Family Foundation report that was through Gallup came out about a month and a half ago. It indicates that teachers are using it to address the material conditions of their labor in K through 12, saving between six to seven hours of labor a week, translated about six weeks of labor a year. That’s going to have a huge effect on our world coming into higher education too, because students are going to have certain expectations about when their assignments are going to be returned. And if teachers are using AI in K through 12 to grade student assignments, they’re going to be expecting us to hand that back to them very quickly. And that deeply concerns me. I was just talking with some of my students here for the first week of class, one of them said that her mother is a 10th grade English teacher and uses it to grade essays. We need to start really thinking about this in terms of how we are approaching equity within our landscape of teachers here at the university level, and how this is going to be addressed on campuses. As Josh mentioned, academic freedom is the sort of law of the land in terms of technology usage here. That door swings both ways, in terms of someone who wants to not use this tool, but then someone who wants to go overboard and start just automating everything. We need very clear guidance, and we really need to start talking as communities about where we think this technology belongs, in terms of that space for our own labor.

Rebecca: I think Marc just opened up the door to a policy discussion around AI use to some extent, because institutions are making some decisions about the availability of tools, putting guardrails in some places and not others, and the entire university system includes a mix of faculty, staff, and students who may or may not be using AI. So can you talk a little bit about the role faculty might want to play in policy development around this topic, or even what it might look like at the individual level in a class when you’re trying to negotiate the absolute use that some students are using AI, and maybe the learning outcomes that you have or the policies that you have in place. Like Josh, I’m thinking about if you’re not allowing the use of AI, but obviously some students are going to use AI, how do you navigate those situations?

Josh: Right, and a lot of that’s tied to my grading strategies of collaborative grading, and so removing some of the incentives to use AI in the first place is a part of that. And Marc has done a lot more at our university in thinking through kind of institutional policy. I’ve been on a committee. Mark, do you want to talk a little bit about this? I mean, my own view is that faculty should absolutely be deeply involved in creating a policy at an institution. But Marc, do you want to say some things?

Marc: We are doing a pretty good job of involving faculty in the actual conversation. It does feel like we are kind of in this endless loop, though, that the main guidance that we’ve established, and also, I think other universities have established too, is it all goes back down to faculty freedom. We’re giving individual faculty members wide leeway in deciding what is appropriate and what’s not appropriate inside their classroom, what’s appropriate in their discipline. We don’t really know how certain faculty are going to be experiencing using these tools to completely automate certain things that I have troubles with because of my values, like, I would never want to automate completely feedback. I would never want to automate grading, or emails even, or letters of recommendation. Those are all things that I value, being a human being with my students and the relationship that we have. It is becoming apparent, though, from some different research that we’ve had too and some conversations with people that’s other than faculty, think very differently about that, and I think we’re gonna have some problems with this. I don’t really know. I know, John, your background is in the economy. I don’t really know many economic examples where some members of the workplace choose to automate the majority of their labor while others don’t, and how that works out long term. So I do think we need some very firm guidance about automation that keeps the role of AI into words of collaboration and augmentation. And really does have that to be a little bit of a hard line about using that to be incorporated so heavily that it starts impacting a person’s day-to-day job, their day-to-day life.

Emily: The only thing I’ll add is that I think students should be a part of this conversation as well, like both on an institutional level and on the individual classroom level. I try to involve students as much as possible in creating my AI policies and letting them choose kind of the pathways that they want to take, especially when they’ve got all these different instructors doing different things in their different classrooms, they don’t always know what’s going on across the university, but students do? Students have a good eye about what’s happening in their class on this side of campus versus the class on the other side of campus, and I think can give us some really good insights. So I want to advocate for involving students in policy discussions as well.

Josh: Yeah, completely agree. Great point, Emily. I was having a discussion with my students on our classroom compact yesterday, and this was one of the discussion questions technology and AI use in education more broadly, but in our class, specifically.

John: From faculty’s perspective, one of the things we hear most often is concern about assessing student work when it may not necessarily be their work. What are some strategies we might use to encourage students to behave ethically, or perhaps to design assignments that are less subject to inappropriate or unethical AI use.

Rebecca: And I’ll add interventions that might need to occur when unethical behavior has occurred, because I think that’s one of the things that faculty are deeply struggling with.

Emily: So one of the things I keep repeating about AI use these days is that we don’t actually have an AI problem in higher ed. We have a crisis of purpose that AI has exacerbated, and AI has made this whole [LAUGHTER] kind of higher education project untenable given the crisis of purpose that we have. So I mean, I think that school is increasingly viewed by students as a transaction. I think a lot of faculty have noticed this, and they’re doing work to get grades, to get a degree, to get a job, and of course, that’s not students’ fault, because that’s what we’ve told them college is for since they were kids. But I think maybe we’re just starting to wake up to all the ways in which we as teachers have inadvertently encouraged that transactional mindset. So I think a good response to AI would be thinking about how we get that out of our classrooms. And so grading systems, I think, are the main way that transactional approach shows up. So if you take grades away, you take away a really important token in that transactional economy, and then what’s left, right? If you take the grades away, well, like hopefully learning is left, and it takes students a little students a little while to adjust to that [LAUGHTER] and to understand what the new purpose is when there aren’t grades. So that’s what I’m always trying to do in my classes. And we could talk forever and ever about alternative grading and different approaches to grading systems that help students move out of that transactional mindset, but if they’re not in that mindset, if the point is not just to create a product to get a grade, if the point is to go through a learning process, and if we incentivize going through a learning process as opposed to just generating a product. In my classes, I’ve had good success seeing less AI use or misuse than one might expect. But the other thing, I think, is just talking to students explicitly about purpose. Why are we here? [LAUGHTER] What are we doing? And one of the things that I really like to cite when I talk about this, Josh and Marc will have heard me talk about this, and maybe tired of it already, but there’s a study that came out about 10 years ago by David Yeager and a whole host of co-authors called boring but important, a self transcendent purpose for learning fosters academic self regulation. And one of the things that they found in this study, or that they show in this piece, is that giving students what they call a self-transcendent purpose for learning, so that’s a purpose that’s about doing something good in the world or helping someone besides yourself, giving students that kind of purpose can help them better persist through tedious or challenging tasks. So I think one of the mistakes we’ve made when we’re talking to students about purpose is to focus exclusively on their self interest. How is this going to help you pass the next test or get into the next class, or succeed in the next class, or succeed in your job or whatever? And I really think we should lean into kind of bigger questions about the purpose of our discipline. What kind of good does the work that we do here do in the world, and how can you use the knowledge and skills that you are developing here to do good things, to help other people? Because I think, you know, none of us are here in purely self-interested motives, right? We don’t get paid enough for that. And I think the same is true of students. We all want to do good things in the world, and so one of the things that I’m really trying to lean into as a way to motivate students and kind of mitigate the temptation of misuse AI, is to talk to them a lot more about purpose and help them buy into that, create structures that incentivize their engagement with the process.

John: We’ll put in a plug for David Yeager book 10 to 25 because it contains a summary of that and much other research done on student motivation. And it’s a superb book to address those issues.

Rebecca: I believe we have a podcast episode on that that we can link in the show notes.

JOSH I would just add two things to what Emily said, I completely agree about the motivation structure that grades bring to courses, we’ve incentivized the use of things like AI when we put all the emphasis on extrinsic motivators. So there was a piece Inside Higher Ed today that was basically a recap of survey results, and there the two top motivations for students to commit acts of academic misconduct or cheating. One was the pressure to get good grades, that was the top, and the second one was short on time. And so there are pedagogical innovations to address both of those things. Emily addressed grades. I would say that there are some alternative grading practices that address the second one as well, the time. But there are other things, like flexible deadlines. There are ways to address the time issue as well that take those top two reasons that students might turn to AI off the table. So that’s one thing. The second thing is, we have a lot of great work, Cate Denial, Peter Felton, etc. many people who are pointing to relationship building, trust building with students. Now, there’s this great facet of human nature that when we feel like we are trusted, we want to be more trustworthy, and so creating classroom environments that really build those kinds of connections and relationships, I continue to feel, will mitigate some of this use as well.

Rebecca: I think one of the things that we see faculty doing, I think, out of desperation, really is turning to AI checkers and things to evaluate or find a way to identify whether or not it’s the student’s real work. And obviously a strategy is to try to mitigate the desire to cheat in the first place, but some people will still do it. Rather than going to an AI checker or something, what are some other ways to maybe address the issues that arise when students do cheat?

Marc: Well, I think cheating is a very loaded term here, too. This is what becomes very complicated, because some faculty have policies that are banning AI in every aspect of learning, which I’ve told them is not really appropriate or acceptable, because you can’t possibly enforce this. And we’re starting to see a lot of students using this on their own time outside of class to work through problems, to research. So we have to really define what is academic misconduct using AI in different terms. We also have to be aware of what the actual technology can do, too. And now multi-modal AI is here, it’s free, the voice in ChatGPT, the actual turn on the camera, it can actually start scanning things around you. That adds more dimension to this than just text, and so we have to be really aware of that, and also the fact that AI agents can now automate quite a few different things in the research process, are all adding to this sort of process. So to me, it’s always going to come back down to a conversation with a student that goes back to openly disclosing how an AI tool or process was used, and bringing that up. And that works really well in a small classroom where I’m supported and have time to do this. It does not work well when I’m teaching a large lecture class, too. If I have 80 or 100 students, that’s where this completely falls apart, and people turn to AI detection services or tools. And there are a lot of them out there right now. In fact, I think Turnitin just released a humanizer detector, because a lot of people have been using AI to humanize the writing, to make it sound more human. And now Turnitin now has a system that supposedly is going to detect when you’ve humanized it, and their other system that they just released too is called Clarity, which is a process tracker that allows you to view students writing long term for days, even weeks at a time. And I’m really against surveillance in this sort of sense, because, again, it’s not really a pro teaching sort of strategy. It also doesn’t really work very well. At the same point in time, I’m very aware that a lot of faculty are sort of struggling with this. And what I tell them, if you do turn to AI detection and you’re starting to bring your students up on academic misconduct charges, you need to be aware of the fact too that there’s very little procedural fairness involved in this. And I do flip that around on them. I said, Look, if you’re going in for promotion and your dossier got ran through an AI detector and somehow came up as positive, what defense are you able to offer it? And I’ll give you a hint, there really isn’t one, unless you’re keeping track of all your documents five or six years ago and how you wrote them. So we need to be really cautious about how we’re dealing with this. And I think the best thing to do is to obviously bring them in and do this. University of Texas has a really great restorative justice program for academic misconduct. Penn has been piloting one for 10 years now, in fact, that’s the second most likely outcome that they have for an academic misconduct is they go through restorative practice to see if they can’t mend the trust that was broken in the classroom before taking a student up on charges.

Emily: Yeah, I appreciate a lot of what Marc said, and I appreciate the question Rebecca, because I feel like sometimes, especially when Josh and I start talking about this, people think that we’re a little like out of touch or something. [LAUGHTER] Because when we’re talking about student motivation and engagement et cetera, some people are like, “Yeah, yeah, yeah. Like, that’s fine, but that won’t solve the problem.” But obviously, as you say, there’s no kind of perfect solutions to this. I think that we should lean into that student motivation and engagement piece, because really, I haven’t seen any kind of viable solutions for mitigating AI use outside of that that I’m comfortable with, but I do think there are room for some kind of assessment security measures, or kind of, however you want to phrase that, as long as we’re varying the kinds of ways we assess students, occasional in class, worker assessments. I think oral assessments are tricky in a number of ways, but may have a place. And I think in terms of what to do when students have misused AI when you’re following up, I think Marc is absolutely right that conversations are the way to go when you can do that. I’m also like Marc, pretty uncomfortable with a lot of process-tracking measures. I tried to do this on my own. I recorded myself writing a blog post, and that totally changed the way I wrote. And I did not have a good time when I was writing, knowing that somebody would see that later. And so I have a lot of mixed feelings about it. One of the things that I do is I do ask students to write in Google Docs. I do tell them about version history and show them how version history works. We talk about this. We set these guidelines together at the beginning of the semester. If I have concerns about your work based on my own reading of it, I might go into the version history to like, look at how your piece came together, and I will ask you questions about why did you write this? What was your process like? Can you tell me what you meant when you said whatever? And again, that’s something you can do with smaller classes. It’s a lot harder in larger classes. So there’s no perfect answer here. And I have a lot of mixed feelings about all of it. [LAUGHTER]

Josh: That’s for sure. There is no perfect answer. And when I hear Marc talk about all the twisting and turning that ed tech companies are doing to try and pile band aid on top of band aid, I guess I just immediately think, if only we use this much time and money to solve other problems in our world of education, that’s ultimately what I think. Continuing my theme for the day, when I was a graduate student and first started teaching writing, it was before Google launched, and so we were taught how to handle plagiarism, was to have a conversation with students, was to ask them about their ideas, to say, “Oh, this is an interesting word. Could you help me understand why you used it in this context?” And that part of what would emerge from those conversations would be either an admission, “Okay, you’re right. I didn’t write this” or it would be a really interesting intellectual conversation about their work, “I used it because of this, and here’s why I…” you know. So I still think that, rather than using machines that may or may not have various degrees of fallibility in this, I think those conversations are ultimately a productive, pedagogical way to handle it. Marc’s right. It doesn’t scale very well, though, so large classes present a problem. In my own classes, I use collaborative grading, which means that I don’t give any grades. Students do a lot of self assessment. I give a lot of feedback. They propose their final grade at the end. I know Emily does this as well. I take grades out of the equation. I take deadline pressures out of the equation by having best buy dates that they can petition to change for themselves if they need to for some reason. If ultimately… I realize I’m an outlier here… if ultimately, even within that context, a student is going to use AI that’s going to be their decision. If that’s what they want out of it, then I’m gonna teach the way I feel like I need to teach, and I’m gonna work with the students in the way that I think will be as meaningful as possible. And people, ultimately, Rebecca, to your point, are going to continue to be people.

Rebecca: I just wanna just note that if there’s a tool that’s detecting humanization, I wanna know like, is the opposite then the dehumanization like, it’s an interesting word choice. I just want to put that out there. But I was just mostly sitting here thinking that’s really like an interesting word choice. [LAUGHTER]

Marc: It is. I do think something for both Emily and Josh that might be helpful to talk about… Grammarly just launched a new product about AI agents that supposedly can actually grade a teacher based off of a rubric that the student supplies from the teacher, and also includes sort of a scan of the sort of public persona of who that teacher is, and gives it a best guess. I have some very personal feelings about this that are not very positive, too, but I wanted to get you guys take on this about how we’re seeing that sort of transactional framework being taken up by other companies as well, too.

Emily: Yeah, I’m not a fan. [LAUGHTER] I mean, there’s so many reasons. The main thing for me is that it keeps students tethered to that grade motivation, right? Like, if you’ve got these Grammarly tools that are gonna guarantee for you, I can help you write an A paper, and I can grade it for you and tell you that it’s gonna be an A, then students are writing to conform to whatever Grammarly tells them an A is. And of course, I think the idea that it would scan the Internet to determine what your instructor might think based on their like Rate My Professor comments or whatever, is incredibly disturbing. And the only good thing that might come out of it is that instructors might get a sense of what it is like to have AI reading you in a way that might make them more cautious about putting their student work into an AI.

Josh: Yeah, I wrote a little bit about this on LinkedIn, and I just want to thank Grammarly for helping Emily and I to make the case that grades are truly subjective and kind of pointless. If the machine can gather some of these random bits of information and suggest what might possibly be a grade that I myself would give, I think that that just explodes the notion that a grade is something that meaningfully measures learning. It’s absurd to me, but I think that that has been a through line here, of a lot of my comments, but that especially, I think, takes it to a new level.

Emily: I guarantee Grammarly does not grade the way that I would. [LAUGHTER] My priorities are not Grammarly priorities.

Marc: I think the big picture we’re seeing, too, is that there’s the performance versus the reality, and that’s what we’re kind of getting back into the overall concept here, where we’re talking about. Teachers are human beings. We’re still flesh and bone for now, hopefully that’s going to remain for this future too. Yet there’s still a huge desire to have that performance. I’m sure Grammarly too thought about this. I’m sure they beta tested it too, because it’s what students want. I want to know what my teacher thinks about this essay at midnight that I’m writing this. So I think we need to start thinking about how we can communicate our value and what we provide to students too, in a way that is better than what a company or a product or just AI in general can offer. And that’s going to be incredibly challenging, because right now at higher education, we are on the receiving end of a fire hose of these different products.

John: We are reaching the end of the time we’ve scheduled. So we always end with the question: What’s next? …which, in anything dealing with AI is something we’re all asking on a daily basis, it seems.

Emily: Well, this semester, we are doing at the Center for Excellence in Teaching and Learning a reading group on the book The Opposite of Cheating: Teaching for integrity in the age of AI, and Marc is also doing a number of AI workshops with our academic innovations group. And to go back to the grading conversation, I think what’s next is I’m continuing to work on a book on collaborative grading, and Josh and I are working on another book on alternative grading, and so we’re not going to explicitly talk about AI too much in those books, but I think they have a real bearing on some of the AI stuff that we’re seeing right now.

Josh: I think an important message to share what’s next is keep doing the work. We’re going to keep doing the work of understanding, learning, and working with our students. I think certainly we will continue to have conversations about AI, but a message from my what’s next, the message that I am continually promoting to administrators here on campus is that AI is one thing among many things that are still at issue in the world of teaching and learning, and so we need to still be addressing those as well.

Marc: Yeah, I think that’s a really great point to end on, too. For me, I’m focusing much more about how multimodal AI is affecting our mental health, in some ways, especially teen mental health. We’re seeing a lot of use cases for therapy and companionship, for these tools that you can talk with. I think that’s going to be something that explodes as a topic over this next year. I also think it is going to be imperative for us to have conversations with our students about how they’re using these tools, because very few people are really talking to them about it and asking them if you are using it, are you thinking you’re talking to a person or to in it, a machine or private company giving all your private information to it? So I think more or less caution is where I’m headed for this with AI, I think we’ve got far too many different products and different use cases out on the market to really keep track of, so I would hope we’d be going to slow down, but I don’t think that’s going to happen anytime soon.

Rebecca: Well, thanks for a really great conversation. We always enjoy talking to all of you, and it was great to have a diversity of perspectives and a more nuanced look at this topic today.

John: Yes, thank you. It’s great talking to each of you, and we’re looking forward to many more future conversations. And I’d also like to put in another plug for the Opposite of Cheating. We’re going to be doing a reading group on that here as well this semester.

Emily: Thanks, John and Rebecca,

Josh: Thanks everyone.

Marc: Thank you guys.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Madison Lee.

[MUSIC]

406. AI: A Student Perspective

In past podcasts, we’ve often explored faculty concerns about student use of generative AI. In this episode, Kaija Hoyt joins us to discuss the evolution of her AI use. Kaija graduated from SUNY Oswego in May 2025 with a major in Human Resource Management and a minor in Mathematics. After graduation, Kaija provided a presentation on this topic to Oswego faculty and staff during an AI symposium.

Show Notes

Transcript

John: In past podcasts, we’ve often explored faculty concerns about student use of generative AI. In this episode, a student describes the evolution of her AI use.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Kaija Hoyt. Kaija graduated from SUNY Oswego in May 2025 with a major in Human Resource Management and a minor in Mathematics. After graduation, Kaija provided a presentation during an AI symposium on our campus in which she discussed the evolution of her use of AI tools. Welcome back, Kaija.

Kaija: Thank you for having me again.

John: Today’s teas are:… Kaija, are you drinking any tea?

Kaija: I am not today because the room that I’m in right now is too small for me to store my tea. So it’s really sad because for this entire month I’ve been tealess. Yeah, it’s very unfortunate.

Rebecca: That is very, very horrible.

Kaija: Yeah.

Rebecca: I have English tea time today, John

John: And I’ve been without tea, but I picked up some tea at a Whole Foods across the street from Duke where I’m recording from, and I’ve got some ginger peach black tea, again, restoring one of my favorites.

Rebecca: Oh, look how it doesn’t matter where you are, [LAUGHTER] you can still drink the same tea. [LAUGHTER]

John: It’s so convenient. So we enjoyed that conversation we had with you in last week’s podcast on microaggressions, and we thought we’d bring you back for another discussion. And once again, this is based on a presentation you gave during a workshop for faculty on our campus, except this time you were talking about how your use of AI tools evolved. ChatGPT arrived on the scene in late November of your sophomore year and was rapidly adopted by many students. What was your initial reaction to ChatGPT and AI tools in general?

Kaija: I hated it. I was so mad that they had kind of become a thing. Because for me, who’s always been like a strong writer and a strong student, it kind of devalued that in a way, because a lot of students were using that, and their writing was about at the same level as mine, or stronger. And some professors were starting to maybe not realize that it was AI doing that, and expect me to write more of that standard. And I also didn’t like the idea of using it. I’ve always really wanted my work to be fully my own, and so when it was first coming about and other people were using that, I was just very strongly against it, and still to this day, to some extent, like, I’m not a huge fan of AI and the use of AI in a lot of environments, but for me, I wish it hadn’t come about when it first did.

Rebecca: I hear, though, that your position or reaction or response to generative AI has shifted a bit over time. Can you talk a little bit about what changed your opinion?

Kaija: Yeah, so I was very firm in the AI use all the way until, I would say, honestly, like the end of junior year, so pretty much like a solid year and a half of being anti AI. I had a professor, Dr. Friedman, who loves AI. He’s kind of obsessed with it in a good way. So in our classes with him, he really encourages the use of AI and gives us some ideas on how to use it. He was really passionate about “You could use it to help you with this part of an assignment.” Or, like, for example, with writing assignments, he would say, “you could use it to draft an outline for you and then build off the outline.” Or when we were doing our compensation project, he said, “help it come up with competencies for you, help it come up with the weights, it can help you with all of that.” So he pushed it pretty strongly. And that compensation class I took with him junior year in the beginning. And even though he was saying that and pushing that, and my group members were like, “Yeah, let’s use it. Let’s use it.” I was like, “Listen, if you want to use it, you can, but I’m not going to” like,” if that’s what you choose to do, fine, whatever. I don’t care, but I’m going to still steer away from it.” And then he also became my honor thesis advisor at about that same time, and that was when he started pushing AI at me even harder. And I kind of was like, “okay, whatever like, just to get you off my back, I’ll try it this one time.” And then I was like, “oh, you know what? That actually did help a lot.” So I started realizing, like, “okay, it’s really useful with things that are, like, the small work that you have to put into a project that doesn’t really have to come from you, like things that AI can do for you that will make it easier to do the project and will also leave more time for you to, like, brainstorm and be creative.” So it takes that monotonous part out of it, which ended up being really helpful, and it also is really good with writer’s block, I feel like, so that was kind of the thing that switched me, for the most part, to being okay with using AI.

John: Could you tell us a little bit about how you used it in your writing?

Kaija: Yeah, so in my honors thesis, I used it when I was having a hard time, like merging two ideas together with like getting it to flow, just having it give me ideas. I very rarely used what it gave me, because basically what I did was like, I would have it help me, and then that would spark an idea for me. Maybe it would give me a transition word, and then I’d be like, I should use furthermore here, that would work really well, or something like that. So it was more of like a jumping off point for brainstorming, but it also helped me a bit if I was struggling to paraphrase something, like getting it to what is another word that fits here? Because sometimes I know there’s a word, and I think I know what the word is, but I just can’t remember it in that moment. So it did help me a lot with that. I also used it with restructuring my thesis. Because when I got to a certain point, I don’t even remember what point this was, but there was a point when I was like, wait a minute, my outline doesn’t make any sense for what this has turned into. So I needed to, like, move things around. And that was really stressful, because at that point I was definitely over halfway done. So there was, like, a lot of paragraphs, there was a lot of information, and I kind of knew how I wanted to move it around, but I also wanted that feedback of, does this make sense? Are these transitions better? worse? the same? And I was trying not to put too much work on my advisors, because I knew they were both very busy at the time and I also really prefer to get their feedback after I know what I want, so that I don’t feel like I’m just doing what they tell me to do. I like to know this is how I want it to be, and then hear their feedback, and then go from there. So I used it to help me switch up that outline a little bit, and I didn’t take exactly what it told me, like this paragraph should go here. For some of it, I was like, “Yeah, that makes sense.” And for others, I was like, “Actually, I feel like this is a better transition,” but it helps me kind of be okay with making that change. So basically, it’s helped me in a lot of ways with writing, but it’s mostly just that starting point and getting me comfortable with what I’m doing, and then later on, it basically looks like AI didn’t help me at all because of how many changes I ended up making.

Rebecca: It sounds to me like some of the ways that you’ve been using AI are to find out what you don’t want, like to bring clarity. [LAUGHTER]

Kaija: Yep, a lot of the time, a lot of it is because there are things that AI wants or thinks that I hate, or just, I’m like, “That is not my writing style. I’m not going to do that.” So I do definitely use it in that way. Or like, sometimes if I give it a paragraph to proofread, and then I see how it wants to make that paragraph, and I’m like, that’s not what I’m trying to say. And it helps me be like, this is what I need to change to make it work. So I definitely used it in that way.

Rebecca: One other thing you mentioned was the writer’s block. And I don’t know any faculty or students that get writer’s block ever. So I’m asking for a friend. How were you using it to overcome writer’s block? You hinted at a little bit, but could you give some more specifics?

Kaija: Yeah. So one of the big things is if, for example, I at one point was going to talk a bit about some psychological theories that relate to microaggressions, and Dr. Friedman had suggested that I cover that at some point in the thesis. He’d given me some brief explanations of different theories he thought would apply, like he gave me a lot to build off of, but I didn’t feel great about writing it, because I’m not really a psychology person, and so I asked AI for some help. One, it helped me find sources, but it also helped me with figuring out how to work that into my thesis and starting the writing. So I told it what Dr. Friedman wanted, and then it kind of gave me, like, “you could do this.” And so then I was able to figure out, okay, from that, what do I want to do? It’s also really good with brainstorming outlines and ideas for projects, because sometimes you just get stuck and you can’t think outside the box at all. Usually, in my experience, like I’ll ask it to brainstorm something, and I won’t really love the answers it gives me or the ideas, but it’ll spark something else. It’s like, if you talk to someone else about it and they gave you a bunch of ideas, except you don’t have to find someone who wants to talk to you about it. You can just ask it, and it’ll word vomit some things at you, and then you can be like, “Okay, I know what I want to do now.”

Rebecca: An introvert’s dream.

John: And also, if you’re doing some work at two or three in the morning, it might be easier to have that conversation with ChatGPT than to wake someone up and ask them to give you feedback. One of the other things you mentioned was that the type of feedback you were getting on your writing was different than the type of feedback that people get from Grammarly and other tools, because with Grammarly, which students have been using for quite a while now, it’ll tell you what to change, but it won’t explain why it should be changed. I think you were getting a little bit more information from ChatGPT in terms of the types of recommendations it would use, and that also ended up affecting the writing that you did later during your thesis. Could you talk a little bit about this?

Kaija: Yeah, so one of the main differences for me personally was occasionally I was looking for grammar checks, but that wasn’t my main focus, because grammar and spelling, like, I’m good on that, and mine is more like, “How can I just make it even better?” So I feel like Grammarly can do that, but yeah, like you said, It never tells you why it’s doing it. I also find Grammarly a little bit annoying, because it just yells at you about everything, and that’s not what I want. Like there’s specific things that I’m looking to fix, and you can’t tell it that, whereas with ChatGPT, you can be like, “Hey, check this paragraph to see if I used also 50 times instead of mixing it with different words.” Or one of the things that I did with it was in my thesis on microaggressions and talking about the trainings that I was proposing, I wanted it to be trainings (plural) consistently, because that’s what made sense for what I was writing. But a lot of the time Google and Grammarly, whatever, really wants it to be singular, which makes sense, but in the context that I was using it, it needed to be plural, even though it thought it was wrong. So I’ve used it in one way with that to be like, “Hey, if you see it, say training singular anywhere like highlight that or call that out so I can fix it.” Another thing, this isn’t a real example for me, but I know some people will start a sentence “although this” and then they never finish, like the last part, so you can tell it look for those situations, and it can highlight them for you. Another thing that I did was there’s like, active/passive voice, I tend to forget to check that, and also really struggle with like when I reread something to check for that, sometimes I just get distracted by something else and I forget what I’m looking for. So I tell it, like, call out those moments, and then I could see it. And then over time, like, the more it was like, “You need to fix this, you need to fix this, the less I made that mistake. So it was really more of a learning thing for me with figuring out, like, “Okay, what are the problems in my writing and what is the way to do it better?” And then, like, I would say by the end of the thesis, like, I wasn’t really using it that much, because most of the problems that I was having I figured out what I was doing wrong and how I should do it better.

Rebecca: Based on some of your examples, I’m thinking other examples that fall into the same category, or maybe, like, checks for first and third person, other things like that that we all make mistakes on when you’re initially writing a draft.

Kaija: Yeah, for sure. I think if it’s one of those things where you maybe are someone who writes… this is really good for students, because I know there’s a lot of students who like to write in the first person… I think it would be awesome if professors who notice, like, “Hey, you’re someone who has a really hard time getting rid of those ‘I’s’ like you love to write. ‘I think that,’ or ‘I believe,’ showing them how to use ChatGPT to help you reword it could be really helpful,” because it’s not that they want to submit it with the ‘I’ in there. It’s just that they don’t realize there’s a better way to do it.” And the more that you have something telling you this is the better way, like, the more likely you are to catch on and to learn it. And a lot of times, professors don’t really take the time to do that with students. They just say, like, “don’t use I,” but students don’t understand how to not do it. So they really need that assistance with the teaching on that part.

Rebecca: One thing, Kaija, that you mentioned earlier was telling ChatGPT, like, “No, I don’t agree with you. [LAUGHTER] I want you to focus on this in a way that you felt like tools like Grammarly couldn’t do that.” And I know that that’s something I enjoy doing too, like telling the machine, “No, do it this way instead. Try this instead.” Can you talk a little bit more about some of the iterations you were able to accomplish using ChatGPT?

Kaija: Yeah. So with different iterations, I would definitely say there were, especially with the structure piece. So trying to restructure it, I think I changed the structure a couple times. And one of the things that I started with, figured out the main ideas of my paragraphs, gave it that, and said, “What order makes the most sense?” And then it did whatever it did. And I was like, “Actually, you don’t realize what the paragraph really includes. So you don’t understand. I don’t agree.” So then I gave it more information. So instead of just like this paragraph defines microaggressions, I really summarized it a little bit deeper. So that was one of the things where I kind of went through a couple different attempts with it and got it to understand what I wanted. Another thing that happened, this is, like, less related to it proofing my writing, and more related to, like, the output it gave me. So one time, I gave it a paper because I had proofread it twice, and I was tired of staring at it, and I knew that there were mistakes, and I just couldn’t find them, so I gave it the paper, and I said, “Don’t edit it,” because I gave it the paper the first time and said, proofread this, whatever prompt I used for it, something along the lines of like, proofread this. And then it gave me the essay rewritten how it would write it. And I was like, “No, don’t do that. Don’t rewrite it. Just tell me what is wrong with it. Tell me what needs to be fixed.” And then I don’t even remember what that output was, but I was like, that’s not really what I want. So then I said, “Okay, without changing anything, just tell me what the strengths of this are, what the weaknesses are, and what things are just clunky.” And then it gave me a very broken down, “These are the parts that are strong. And then a couple of quotes of examples. And then these are the areas that could be improved.” And then if, with the improvements, I needed help with, like, how do I do that? Then I can ask it that, like, “Well, how do I make this more concise?” And it would give me examples.

John: When you gave that presentation to faculty, one of the other things you talked about was using this to help prepare you in your math classes. Can you talk a little bit about how you use ChatGPT to support your learning in math.

Kaija: Yeah. So it was in my introduction to algebra class, which started off a little rough for me. I was having a really hard time paying attention in class for the first, I want to say, like a month, and that first test came around, and it’s not go well at all, which is very strange for me, because I usually do really well in math. So that was, like, a big blow for my ego. And I was like, “Yikes. Okay, this is a problem.” So I started paying more attention in class, and that helped. And I was doing the homeworks and starting to understand them, but the first stuff that we had learned, like, kind of I knew, like, “I just really don’t get this, and I don’t know if I’m going to if I just keep going in the class.” So we had a textbook, but it was one of those textbooks where it’s like, the professor uses the textbook, but not really, so it doesn’t directly match up. It’s not like Calc I and II, where you are literally doing your homework out of the textbook, and the textbook follows the same path that the class does, like this class in the textbook were different enough that it made it kind of challenging to find practice problems in the textbook. And the other thing is that practice problems in the textbook are not always what you’re looking for, and there’s no way to like, filter them out, like, you just have to stare at it and find the one you want. You can’t be like, “Well, I’m looking for how to do mod eight.” You have to find that in the massive answers that there are. And the professor did give homework, but I think we had like one a week, and there was only like six to eight questions each. And I’m very used to math classes where you have like 20 questions for a night at minimum, although those are usually optional, but I do them because I love math. So it was kind of odd for me. And I was like, “Why are we not doing more?” And I also, in addition to the fact that I was struggling, like I also wanted to do more with it, I really enjoyed it. So I went to ChatGPT, and I was like, “I feel like this could help me.” So I kind of told it what the class was, told it what I was learning, and then said, “Do you understand what this topic is?” And it was like, “Yes, I can give you practice questions.” So then I would tell it, “Okay, can you give me a question on Z/8?” and then it would give me something, and then I could do it. This is the one that I had a problem with. It was that I told it like, “I just want you to say whether it’s right or wrong. That’s it. Give me the question, I’ll tell you the answer, and you say ‘yes’ or ‘no,’ and then if you say ‘no,’ then I’ll be like, ‘Okay, walk me through it.’” But it just kept giving me the whole thing anyway, even though I told it, like five times, so I just gave up. So that was kind of how I used it for mostly getting practice problems. And there were also sometimes, if there was a concept that I was struggling with, I would have it kind of walk me through it. And honestly, it’s pretty impressive how well it did with this class, because this is like a higher level math class. I did not expect it to understand it, but it really was good at walking me through like how to solve it. And then from that, I was able to create my own understanding of what to do with some of these problems.

Rebecca: Kaija, what class was this in?

Kaija: Introduction to algebra, which sounds easy, but it’s Math 330 and it’s not like solving simple algebra equations. It’s like abstract algebra concepts.

Rebecca: I kept thinking that doesn’t sound like intro to algebra. [LAUGHTER]

John: When you were getting these problems, one of the things you noted when you gave a presentation describing this was that you learned how to modify your prompts a bit to get more useful feedback. Could you talk a little bit about what sort of modifications you made to get more useful results from ChatGPT?

Kaija: I, for one, kind of learned how to get better at telling it what kind of problems I wanted, because I started off with being like, “Can you give me an example of something like asking whether or not it’s a homomorphism?” I gave it no context, so it didn’t know what types of groups we learned about and what types we didn’t. So one time it gave me this group that I honestly was like, “I don’t even know what that is. Never heard of it.” So then I was like, “Okay, I’m gonna start telling it like, these are the groups we know. Give me an example from there.” So that was for the practice poems it gave me, and then when it gave me the feedback on whether it was right or wrong. If it was a time when I was like, “Okay, I actually need this feedback because I didn’t do it right or I could just then I would read through what it gave me,” and then I would give it follow up questions of, like, “Okay, here’s where I’m stuck with your explanation, L I don’t know what this means.” Or “Can you clarify this more,” which sometimes it would just regurgitate the same thing. So then I had to figure out a different way that I can phrase this to get the part that I’m confused about out of it.” So asking, like, “Okay, I don’t understand where this Z came from. Why is there a z here?” instead of just, like, “What does this mean?” because if you just say, “What does this mean?” it’s just going to throw the same thing at you, and it’s not going to help.

Rebecca: So I’m kind of wondering, given the experiences that you’ve had in using ChatGPT in some really different contexts, how would you like to see faculty talk about AI in their classes, or what parameters or ideas do you think would be helpful for other students to understand how it might actually be a productive way of learning in a class?

Kaija: I have very strong feelings about that, so I’m glad you asked. For me, I think that it was really helpful having a professor who said, “This is how you should use it. This is the ways that it can help you in this class specifically, and I’m not going to penalize you for using it in those ways.” That really is what helped me, like, be open to using AI. And I know, not for me because I was anti-AI, but for other students who can’t seem to get away from using it, when a professor is just like, blanket no AI policy. You’re not allowed to use it like they just decide to entirely ignore the AI policy because they don’t care. They don’t know how to learn without it. And they’re so dependent on it that if you don’t provide them a way to use it productively and without plagiarizing, then they just can’t move away from it, and obviously some students are always going to use it to write their papers for them. I’m not saying that this is going to solve that problem totally, but I think that there are a lot of students out there who, if you say to them, “Hey, I’m okay with you using it to make the outline,” or “I’m okay with you using it to help you proofread,” or something like that, and showing them like these are the ways that you can use it without it becoming an academic integrity problem, then they are going to be more likely to actually put that work in alone, because a lot of it is like, they just don’t know where to start. They don’t know what to do. They’ve got writer’s block, and there’s a deadline, and so they’re just like, I’m just going to have it write it for me. But I don’t know for sure, because I wasn’t super close with the people that were in my senior seminar HR class, but Dr. Friedman said in that class for our paper that we had like “You can ask ChatGPT for an outline, and you can build off of that. You can ask it for assistance in writing this paper. You can ask it for sources.” And I’m pretty sure that a lot of students in the class really took that and did that and used AI responsibly and still submitted mostly their own work, whereas with other professors, I noticed them just kind of having it write it for them, or things like that. And I know that, like a lot of the time in the tutoring center, if I have a student come in for writing who is clearly someone who’s really dependent on AI, I try to give them that overview of like, if you use it in this way, A) you’re safe and B) this will help you in the future as well. And it seems to work. So that’s a small example. So I don’t really know, but it does seem to be really helpful to be clear about like, these are good ways to use it, these are bad ways to use it. Instead of just saying all of it’s bad, you can’t use it at all.

John: So during your time at Oswego, I think you had five semesters during which AI has been available to all students. How often, though, did professors talk to the students about whether AI could be used, when it should be used, and why it might be inappropriate to use it in other areas? Was that a conversation that happened very often?

Kaija: So I would say I had the one professor who talked about it lot. I had probably one or two professors who really consistently called out, like, please do not use AI in my class. If I catch you using it, I will fail you. And then I would say it’s hard with math, because math professors don’t really talk about AI very much, which is mostly because it can’t really do that much with math, because it’s really easy to catch it, because it hallucinates a lot when it’s math, and honestly, it will take more of your time to use it in a math class to, like, do your homework for you if you’re trying to get a good grade. So they don’t talk about it a ton, but they do say they have their policy of don’t cheat, and that includes AI use, things like that. And then in my other classes, I think most of them had it in the syllabus, like you are not allowed to use AI in this class, that that would lead to a failing grade and being reported to the dean. But if they even said anything about it, they only addressed it, like on the day that they went over the syllabus, and then they would just not talk about it again. So I think a lot of them, like, just put the policy on there because they feel more comfortable with that and they don’t really want to have to address it. That’s kind of the vibe that I’ve gotten. So I would say it’s like, mostly just this thing that, like, you know you’re not supposed to do, but you also don’t really know what’s going on with it, at least in the School of Business.

Rebecca: You’ve talked a bit about math and you were just pointing out the hallucination issue with math problems, and because you were doing practice problems and things with it, how often did you find the hallucinations bubbling up in your use?

Kaija: So surprisingly, it didn’t happen as much because it was more conceptual things than computational, but there were a couple things that were computational that I used it for, and that’s when I found it hallucinating. One of them was I was practicing the product of cycles, and I’m pretty confident in this, but I just wanted to practice it because the final was coming up, and you never know. So I was going through, I think it was the second or third example with it, and I did the multiplication the way that I knew was correct, and I got the answer that I knew was correct, and I told it, this is what I got. And it said, that’s wrong. So then I looked through the output that it gave me of it’s step-by-step, and it’s step-by-step didn’t make any sense. I don’t know what it was doing. I don’t know what happened, because it was working fine before that. I do suspect that it’s possible that it had been right when I’d run out of the like higher-level questions, and it switched me to the lower level, and that’s why it happened, but I don’t know that for sure. And so I got to the end of it, and I was like, “No, I’m definitely right. I don’t know what you’re doing, but it’s wrong.” So I basically told it that. I said, “Your answer is wrong, my answer is right. I don’t know what you did, but that’s what it is.” And so then it responded to that, and was like, “Yeah, you’re right. You got the right answer.” But then, like, it’s calculation, it had the answer, it had before, and then it had my answer, and then it said, “Yeah, so you’re right.” It was like, 123, so 345, and I was like, [LAUGHTER] that doesn’t make any sense. So I did stop checking my answers with it at that point with those because I was like, you don’t know what you’re doing, but that is the only time that happened with the higher-level algebra. I have seen it do funny things with lower-level math classes, like regular algebra or calculus, because I’ve used it for fun with that, just to, like experiment and see if I can get it to hallucinate, because I think it’s funny. [LAUGHTER] And it definitely has happened a couple times with that, which is a little bit interesting, that it can just kind of end up in places. And I’ve also found that if you tell it like it’s wrong, it doesn’t even care if it’s actually wrong, it just decides that it is. And so that is probably part of the problem, is people being convinced that they’re right, and so they’re training it that this other thing is true and it’s not. So it’s pretty confused on things like that. And I think that’s the other thing that I think is why it worked better for introduction to algebra, because I don’t think there’s a lot of people going to ChatGPT for that. So I think it really just had that database of whatever it found when it was first started, and less of that user input that can make it more incorrect.

John: One of the things you described just a few minutes ago was using ChatGPT to get more practice problems, especially in classes where there’s not enough practice problems to make you feel comfortable. Is this something that perhaps students should do more regularly or more often? Use it to help develop skills when they’re not getting practice problems that are relevant to their own areas. When instructors give assignments with a group of problems they’re generic problems that are designed to address the needs of all students, but every student comes in with very different backgrounds in terms of what they understand and what they don’t. Were you able to use this to help develop those areas where you weren’t as comfortable with and not having to spend that much time on those areas that you were more comfortable with?

Kaija: Yeah, so it’s totally adaptable to what I needed, and with what I said about I fell behind because I wasn’t with it at the beginning. So it was something where it’s like, we’re not going to have more practice problems. We left this way back there. It can still show up on the final but we’re not talking about it anymore, whereas he’s still giving us practice problems for whatever we’re working on now, and so I would do that homework, but that was something that I was like getting. So instead of spending extra time on that, I would just go do the areas that I needed, and I didn’t, in this context, but you can have it figure out on its own what areas it is that you’re struggling with, and then it can just throw those questions at you, because you don’t have to tell it: “Give me a question on homomorphism.” So you can say, “Give me a question that would appear on an introduction to algebra test.” And for the most part, it’s good, especially the more information you feed it. And then if you work through, and you work through, it starts to catch on, like, this is where you’re struggling, and this is where you’re set. So it can really modify to you in the same way that your professor would at office hours, without you needing to go to office hours and figure out that schedule with them, because our times usually don’t match.

John: When you presented this, a math instructor asked, “Why didn’t you just go and talk to the instructor with questions?” And you gave a really good response that I think would resonate with a lot of people. How did you respond to that question?

Kaija: So I basically said that for me as a student, I’m a very independent learner, and I also don’t like to ask for help, and so going to office hours is really difficult for me from that side of it, and I personally am someone who would rather spend two hours on a math problem than have anyone or anything help me with it, like I really want to figure it out on my own. That’s something that’s really important to me, and so the more help that I get with it, the less satisfied I am when I solve it. And one of the things that I really like about AI versus a human in that is like with your professor, they don’t really know what pace you learn at, and they don’t really know what specific spot you’re stuck on. And so with a professor like they might just either show me the whole problem or do it in that way where they’re trying to get me to figure it out, which are both great methods, but it’s not necessarily what I’m looking for in that situation all of the time. And so with AI, what I can do is I can do my steps and say, “Okay, this is where I’m stuck,” and then just see what’s the next thing. Or, like, “give me a hint on something small,” which professors sometimes are really good about that. The professor for this class was actually great with that. There was a couple questions that I went to him with, because I’m not going to use AI to solve the math problems. Also, it doesn’t know what it’s doing. So I would go to him and be like, “I’m really stuck, can you give me a hint?” And he would do that, which is great. But when it comes to a practice problem, I don’t want to go in and be like, “Can you give me a bunch of practice problems and sit there and solve them?” So there was that. I also feel like, for me, with the way that my schedule has been because I’m an overworker, so I tend to be like, overloading on credits, working two jobs and running a bunch of student organizations. I don’t have time to go into a professor’s office hours. I could send them an email, but a lot of the time, I find that communication to just be a little bit ineffective. We’re on different time schedules, so when you reply to me, I might not be able to reply for a bit. So there’s that side of it too, where, like, I’m not available during your office hours. I’m honestly not available when you’re awake, most likely. So I can’t really make that work. And so a lot of the time, it’s easier to have something that’s available whenever I have that five minutes to work on a problem, instead of like, “Oh, I have five minutes, let me email them and see if, in this five minutes, they can give me practice problems.” So there was that as well. And then, especially for students like me who struggle with anxiety, going to talk to your professor is like, one of the hardest things that you will ever have to do in college. Like, it’s just really intimidating. And if you don’t feel close to them already, it’s really hard, at least for me, to go into their office and have that one-on-one interaction. With professors that I bonded with throughout the class, throughout the year, like at the end of the year, sure, I’ll go to your office hours, but at the beginning of the year, when I don’t really know you yet and I’m not comfortable around you, that’s just too difficult for me, and so having AI there as a backup and as an extra resource is really helpful.

Rebecca: Thanks for sharing those. I think those are hard things sometimes for students to articulate and say, but I think it’s a common experience for a lot of students. I think one thing that you’ve pointed out, and I’m really curious how you got there, was being able to fully articulate what exactly you needed. So because you were able to say, “this is what I need” to the tool you could get what you needed. But sometimes people aren’t quite at that point, and you hinted at maybe there was times when you weren’t quite sure what you needed. Can you talk a little bit about how you develop some of those skills to be able to articulate what you need?

Kaija: Yeah, so I like to say that I’m a pretty self-aware person, so that certainly helps, like, I am a little bit better at being like, “Okay, this is an area that I’m struggling” or I know what I need right now, but there are obviously times when I’m like, “I have no idea.” So sometimes I had AI help me with it, like I would tell it, maybe just throw things at it, just like word vomit into it, and then have it be like, “Okay, this is what it looks like your problem is,” because it somehow can pick that out. But also sometimes it was like, just using my resources, so looking back through tests or the homework and figuring out where did I get stuck there, and really diving into it and thinking deeply about like, “Okay, I know I can’t solve this problem, but why can’t I solve it? What is it that I’m confused? Do I just not know the definition? Or is it I’m halfway through the problem and I just can’t get to the next step?” So it’s really a lot of kind of talking to yourself. For me, I do that out loud, and it really helps. But just like, having that conversation with yourself about, like, “Okay, let’s really think about this problem, and let’s really think about what part of it is confusing,” or “Let’s think about this class as a whole, and why are my grades not as high as they should be? Where is that problem?” So, like, part of it for me, like I said before, was really just that I wasn’t paying attention in class, and that is something AI can’t help me with. AI cannot help me study for a class if I don’t even know what’s going on in the class. So that was something that I had to be like, “Okay, this is a me problem, and I need to figure out a way to fix it so that then I can use AI as a supplemental resource.”

John: One of the things that’s been in the news a lot are student and sometimes parent reactions to faculty using AI to generate questions, problem sets, and other things. How do students react, in general, to faculty use from what you’ve heard from other students and your own reactions?

Kaija: Honestly, I haven’t heard a crazy amount about it. I think that for the most part we get it. We’re like, “hey, whatever.” I mean, as long as you’re still teaching us, as long as we’re still learning like, “it’s okay that you’re using it,” but where we get stuck is, “how come you can use it to teach us, but we can’t use it to teach ourselves?” That is the part that bothers me. I’m totally okay with you using AI to help you generate the syllabus, generate the questions that you’re going to ask, the learning objectives, like, that’s fine. I don’t really care. But if you can do it, then why can’t I? And to me, it makes that gap between us more. I understand that the professor is above me and standing like, that’s fine. I’m okay with that, but I want, at the same time to feel as respected by them as I’m supposed to respect them. And so when you say, like, “Well, I can use it because I know how to use it responsibly, but I don’t trust you to use it responsibly.” That’s where I kind of have a problem, where it’s like you need to respect students in the same way that you would expect them to respect you. So if you want your students to be okay with you using AI to assist you with the learning objectives, the curriculum, whatever it is, then you also need to be okay with students using it to assist them.

Rebecca: Well Kaija, this has been really great, but we always wrap up by asking, “what’s next?”

Kaija: Yeah, so who knows? So, as I said on the last one, I’m moving to North Dakota. I do have an apartment set up. So moving in on the 15th of August, which is exciting, still looking for a job, which is terrifying. Currently, hopefully I’m going to have one. I’m going in for like, the third round of interviews. So that’s hopefully a good sign with one company, and otherwise, I’m just really, really hoping to do something in human resources out in North Dakota for a couple years and then hopefully come back here.

John: Well, it’s been great talking to you, and we really appreciate your willingness to join us for these two conversations. And we hope everything goes well.

Kaija: Thank you.

Rebecca: Yeah. Best wishes on your job search.

Kaija: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

399. Improving Course Design Using AI

Generative AI can increase efficiency and support student learning, however students can also use it as a substitute for learning. In this episode, Nathan Pritts joins us to discuss ways in which generative AI tools can improve course design and strategies to encourage students to use AI tools ethically and responsibly.  Nathan Pritts is a Professor and Program chair for First-Year Writing at the University of Arizona Global Campus. Nathan’s recent work has been focused on the relationship between AI and human teaching.

Show Notes

Transcript

John: Generative AI can increase efficiency and support student learning, however students can also use it as a substitute for learning. In this episode, we explore ways generative AI tools can improve course design and ways to encourage students to use AI tools ethically and responsibly.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Nathan Pritts. He is a Professor and Program chair for First-Year Writing at the University of Arizona Global Campus. Nathan’s recent work has been focused on the relationship between AI and human teaching. Welcome Nate.

John: Thanks for having me, Rebecca, John.

John: Good to talk to you. Our teas today are:…? Nate, are you drinking tea by any chance?

Nate: I am indeed. I’ve got this Paris tea by Harney and Sons. It’s a fruity black tea with bergamot oil. It’s very nice.

John: We have some of that in our office. It’s a favorite of the current Associate Director of the teaching center here. So we keep that stock pretty regularly. And Rebecca?

Rebecca: I have Golden Monkey today, John.

John: Is it golden?

Rebecca: It has golden tips, yes.

John: Okay. And it’s a black tea?

Rebecca: And it’s good. It’s a black tea.

John: Was it one of the monkey-picked teas?

Rebecca: I don’t know. Maybe it was AI picked. [LAUGHTER]

John: And I have a spring cherry green tea. One of your favorites.

Rebecca: Oh, yeah, definitely my favorite, with those cherries.

Nate: Nice. [LAUGHTER]

John: It’s an ongoing thing.

Rebecca: Yeah, I hate it. [LAUGHTER] Oh, okay, you actually don’t like it.

Nate: No, I actually super hate it. [LAUGHTER]

John: I did end up with a couple of bags of whole leaf tea because she bought some because it smelled so nice.

Rebecca: It smells so good, but I just cannot stand the taste of it.

Nate: Easy to get taken in by some of these.

Rebecca: It’s true, the wafts are so good. [LAUGHTER] Alright, well, we invited you here today not to discuss tea but [LAUGHTER] the impact of AI on the design of online courses. In your March 19, 2025 article in Faculty Focus you discuss some of the ways AI tools can be helpful in augmenting our work while still maintaining human connection in our courses. [LAUGHTER] Can you describe some of the ways in which AI can help us create more effective courses?

Nate: I can talk about some of the ways that have worked for me. I mean, I think one of the things to realize in course design, some faculty will have a lot of people supporting them. They’ll have colleagues to bounce ideas around with, but at other times, a subject matter expert won’t have anybody there to support their build. And I feel like that’s where AI can really help support the course development. It sort of works as just another pair of eyes on the work that the faculty member is doing. It works to bounce ideas around. And I feel like there are some very targeted ways that that can come into play. Always, I’m trying to think about AI as a support for the human intuition, the human experience, what we bring to the table. Ethan Mollick, who’s a pretty well known AI researcher, talks about the human in the loop, which is the person who’s helping AI accomplish goals. I like to think of it the other way around, as having an AI in my pocket. So I’m the human. I’m always going to be the human doing the work, but I can turn to an AI tool to help me do things that I might not be able to conceptualize myself. I kind of break curriculum work down into two different categories. This is kind of what we came up with internally to help us think about some of the different ways that AI can support course design to make effective courses. We think of them in terms of accelerators. That is an AI that’s basically helping a faculty member get started, move faster, maybe streamline some of the processes. A faculty member is staring at a blank page, an accelerator might help them. We’ve developed some prompts that will do that. But then there’s also what we think of as multipliers. This is basically to help enhance the quality of the work that the human, the SME, the faculty member, might have already come up with. And that’s what that article, Rebecca, that you referred to, that’s what that really talks about. The stress tester for assignments is basically a multiplier. It’s assuming that a faculty member would have developed on their own an assignment for a particular class. And then will use AI to essentially test it, run it through some battery of tests just to see if there are ways to improve it. One of the ways AI can do that in this particular case is that it can simulate student perspectives. So you can feed the assignment that you developed, or the discussion prompt, whatever it might be, into AI, and you can ask it to simulate a variety of student responses. And you can give it some parameters. You can say, I want some A level responses. I want some C level responses. You can ask it to mimic what your student population might be if you’re teaching first year versus if you’re teaching graduate level. And I think that can kind of help to show maybe where the prompt could be strengthened, maybe students might misinterpret a particular aspect of it, or maybe they struggle with some part of it. We’ve all had that experience where we develop an assignment, we give it to students, and they just find ways to break it right out of the gate. We never thought to tell them not to do a certain thing, or we didn’t make it as clear as we needed to. This gives us an opportunity, working with AI to potentially catch those stumbling blocks before we put it in front of students. So we refine the prompt, we find blind spots, and hopefully this is a way of making that assignment prompt that we’ve developed a little bit more ready for the wild. And let me just say I get that an AI-simulated response to a prompt is very different from what a student is going to say. Students are going to find new and exciting ways to respond to our prompts. The AI doesn’t work like that, but if we ask it to generate, say 50, say 100 responses to a prompt, don’t share them with me, just generate them and then thematically, tell me what you’re coming up with. It still gives us a baseline. It gives us a sense of where our language might be unclear. Maybe we need an extra bullet point to clarify that we don’t only want a thesis statement, but we also need an entire introductory paragraph. Maybe we need an extra explanation for what essay structure needs to be used in this prompt. We can’t just assume the student’s gonna know. We gotta make sure to put that in. So this is just one of the ways I think that AI can really help in the course design process. Again, I mean, I feel like you’ve got that human developing the material, but then you’ve got the AI as that second pair of eyes, that different perspective, something outside of our own head, that can lead us into areas we might not have gone into on our own. I feel like we’ve all had that experience too of you’re developing a class, you come up with an assignment, and you just love the way this assignment works. You’ve used it for 5 or 10 years. You think it’s fantastic. We fall in love with these things, but students change, and the way that we interact with the content or the subject matter or the outcome might have changed within the course. It might have changed institutionally. And we need to be aware of those things, and we need someone to help us fall out of love with our assignments and really test them. And again, I think in the absence of a colleague who might help us do that, for people who are working in silos or people who might have tough deadlines, AI can really help with that

John: In that article that you just referred to, you also provide an example prompt, and we’ll include a link to the article, and then our listeners can go and take a look at the prompt themselves. And I think you provided an example of using a personal narrative assignment in class, and you also talked a little bit about how you can use the AI tool to refine your prompt based on the responses that it has provided. Could you talk a little bit about that example and how you might use AI to directly suggest improvements on the prompts?

Nate: Yeah. So I teach mostly first-year courses, first-year students. I’m the Program Chair for our comp sequence, so I oversee students who are coming through both of our composition classes. In our writing courses, one of the ways that we handle the fact that these are students coming in with a large degree of fear, a lack of confidence, is we try to meet them where they are in terms of their own skill set, of course, but also in terms of their own interests and ideas. So yeah, it’s a narrative prompt that we try to work on in these courses. We try to get students to look at their past experience and to reframe it in a way that’s going to help them see their academic goals, their lifelong learning goals, let’s say. So we come up with this prompt, and it’s a prompt that asks students to talk about an experience they’ve had. Now I can ask AI to run some simulations, and what might be revealed… in fact, when I tried this, one of the things that was revealed… was that the way that I had worded the prompt was allowing students to talk about emotional experiences, but not those that had any real direct correlation to their choice of major, their choice of career field, and that’s what I wanted. I wanted students to find ways to talk about, essentially, their ethos, how they’ve developed this in a way that helps to see their chosen career field in a different light. My prompt wasn’t doing that. It was allowing students to just talk about very traumatic or sad or happy experiences they had at any age. There was no way of framing that into what I wanted. I assumed students would do it, but they might not. So it helped me to really understand that, okay, if I’m going to ask students to talk about a personal experience or a defining experience, I’m going to have to add material to the prompt that clarifies that I’m not just looking for a personal experience, I’m not just looking for a particularly emotional personal experience, I’m looking for a defining one that led to their choice of major or career field. Most of my students are non-traditional in the sense that they’re already working so they already have a chosen career field. And this idea of major versus career field, we typically think of it in terms of career field. But my point is that the AI really helped me to see that one of the ways in which I could better this prompt was to put a little more scaffolding into it and slant it toward what I wanted. Of course, I was worried, as the classroom teacher, when you’re teaching comp, when you’re teaching any class, it’s not about right answers. You kind of want to be surprised. You don’t want only one type of paper or assignment to come to you as a teacher. So you’re trying to write a prompt that leaves a lot of leeway, let’s say, for students to interpret and to give you unique responses, but what I had done was left it so wide open that they weren’t even meeting the guidelines, reall,y weren’t even doing what I wanted, and so AI was able to help me nip that. Now I could have just put that in the class, and after a section, after two sections, I would have figured that out, but I think that that would have been challenging for those students. Let’s say this is a week two assignment. Well, when week five rolls around, suddenly they don’t have that stable week two material to build upon. And so we’re really looking at helping improve course outcomes. We’re looking at making sure students are learning what they need to learn in the class. And again, AI is an avenue to do that. I hope that kind of explains what I’m talking about. I think there are a lot of different ways to do it in a lot of different classes, but that’s just one sort of avenue I tried there.

Rebecca: It’s a really good example of how the AI tool can do analysis of the assignment. Can you give some other examples of the kinds of holes that AI might be able to identify for faculty?

Nate: One thing I’ve been working on recently is universal design for learning. So I teach at an all online school. Our courses are asynchronous, and as a result, access and equitability, and of course, UDL principles are important to all of us, but they’re very much a part of how we design our classes. But so many subject matter experts are not curriculum designers. They’re maybe not even teachers. They’re experts in their field. And so the model now for a lot of universities is just to have these experts, these subject matter experts, design courses. And they’ve got a lot of great ideas, but they don’t understand some of the basics of instructional design. So we were able to develop an AI-based prompt that helped ensure that UDL principles were baked into materials of the course, while at the same time explaining those principles to the subject matter expert that’s working on them. I feel like so many AI tools are kind of like a black box. You ask it a question, and it gives you an answer. And I think that’s one of the problems we see with student use. They have an essay prompt, and they tell AI to write it for them, and AI does, and then they’ve got a product. They didn’t learn anything. It’s the same thing in course design. We want faculty to understand some of these underlying principles. And so back to this idea of UDL principles, we can create a prompt that says to any AI tool, “Okay, here are the main principles of UDL, and here’s some background information.” We could even give the AI some background research studies. Maybe your AI tool has access to the internet and can find some of these things on its own. You can then check and make sure it’s understanding of UDL is, in fact, correct, and that it’s applying it appropriately, through some refinement and testing. But what you get then is a tool that allows a faculty member to say, “Okay, hey, look, I’ve got a discussion board prompt, and I want to talk about essay structure. And here’s how I’ve typically done it: I ask students to identify aspects of essay structure and talk about how meaningful they are, but I want to think of some new ways to do it, and I want to make sure that I integrate UDL principles into this.” And so now the AI can turn back to the faculty member, not only some different ways to approach that course content element in that format… you’ve identified the course content you wanted, which is essay structure, you’ve identified the format, which is discussion board… and so now the AI tool will give you some options that you might not have considered otherwise, but it can ensure that it’s foregrounding or emphasizing discussion prompts that will emphasize Universal Design for Learning principles, and it will explain those connections. So the faculty member will then have an example discussion board, and it will say the reason why this adheres to UDL principles is because of these reasons. So the faculty is learning right along with it, and they’re coming up with interesting material for their course design. Now they might not use that. They might use that as a baseline to spring off and develop their own material, but maybe they’ll internalize some aspect of that, and they’ll learn while they’re doing it. I think it’s a really interesting moment. We talk about how students need to learn how to use AI. I think faculty are in the same boat. There’s a lot of fear, there’s a lot of uncertainty, but the more everybody interacts with it, the more they can learn about the strengths and also the limitations, and maybe they can teach themselves a few things along the way that they might not have been able to pick up otherwise.

Rebecca: Sounds like a good opportunity to model how to use it as a learning tool, which might be really good for faculty to see.

Nate: And I think that’s how it translates into students as well. I mean, again, if we look at some of the popular sites that allow students to upload their essays for automatic grammar checks. I’m not going to name any names, but one of them starts with grammar, and I feel like what happens is that tool just changes the student’s work, and the student doesn’t really understand why or how what they wrote was wrong or less than correct… I don’t want to say wrong… because there’s no upskilling involved. And so I feel like, whether it’s a student application, or whether you’re using it internally for faculty development, baking in some kind of upskilling is what really is going to make this seem more valuable to people, even those people who are sort of anti AI. I think it can show the efficacy of the process, even if you’re still skeptical of the product.

John: You mentioned that a lot of students will just submit work based on the prompts that they receive. And in a September 2023 Faculty Focus article, you talk a little bit about how some faculty may choose to provide AI tools with rubrics or other evaluative criteria from their courses and use that to provide some feedback on student work. And there’s a lot of concern among faculty about whether we should be using it to grade student work, the possibility that we may have students using AI tools to submit work that we then use AI tools to grade, and there’s very little thinking on either side. And you suggest in this article, we should focus on the uniquely human attributes that faculty bring to their classes. Could you talk a little bit about how we can bring uniquely human elements into our teaching, particularly online teaching?

Nate: We all hear this. There’s a lot of talk about the things that AI can do well and the things that it doesn’t do so well. And I think one of the traps we catch ourselves in is AI is really efficient and it’s really fast. And so a lot of people, faculty members, are thinking, “Well, I could never be as fast as AI, I could never be as efficient as AI.” And so they start to feel pretty disheartened about all of that. And it seems like every day there’s more and more that AI is doing well, maybe it’s not doing it well, maybe it’s just doing it well enough. And so it’s pretty daunting. I think it’s a little scary at times, and I feel like a lot of what I was talking about in that article, and a lot of what I still believe, and when I talk with faculty, this is how I try to bring it up, is just that I feel like we do need to lean into these elements that are uniquely human, the things that only we can do in the classroom. For me as a teacher, my attention, my attentiveness in the classroom to the student work. Those are things that AI can’t hack, but AI can hack feedback. It can give really good feedback. So how is my feedback different or better than what the AI might provide? AI can provide their feedback 24 hours a day, whenever a student wants it. I can’t do that. I’m gonna check my email three or four times a day. I’m gonna get back to students within 12 hours, but I’m not right there with them all the time. I can’t be faster, I can’t be better, I can’t be always on. So what can I do? And I feel like that’s where talking in that article about the idea of trying to find those human handprints. This is something that Kevin Roose talks about and those hand prints, I think, are those elements in the course, those elements of interaction with students that are something unique that we could do. And I feel like that’s something that each faculty member, each teacher, needs to kind of come to grips with on their own. As I shared in that article, it was a pretty harrowing process for me, because every time I came across something in my class that I was like, “Oh, this is purely me. Nobody could copy this,” suddenly, AI can do that. I write snappy headlines for my announcements and I write amazingly emotive and snazzy emails. AI can do those things. So it was this moment of realizing, like, okay, all these things that I thought were conveying my attention and my attentiveness in the classroom to my students, those things are something that AI can do. So what’s left? And the thing I settled on, and it’s a pretty basic type of thing, is to interact with students using video feedback for their essays. I think what might get lost in the article, or maybe what I only came to realize more recently, is that the feedback I’m giving is more developmental. It’s experiential. It’s not meant to say, “Hey, your thesis was wrong, and I’m just going to use my face and voice to tell you that.” Really what I wanted was for them to see my face and hear my voice as I read through their papers. They got to see how I reacted to sentences that they wrote. They could see if my brow furrowed or if I kind of stifled a laugh or whatever. They could see when I was stuck. They could see when I was excited. They could register all that in my sort of running commentary on their essay. And I feel like that was one of those moments where I realized, okay, this is it. This element of connection is what I can uniquely contribute to the class. This is what I’m doing as a teacher in an online, asynchronous environment. It’s very different if you’re in front of a classroom of living, breathing students, but in my environment, that, to me, seemed like a moment of connection, and that became kind of a baseline for me as I looked throughout other aspects of the course. What are some things I can do that only I can do? Or that by me doing them, they become more meaningful than if AI was doing them. There’s an example in our courses. It’s an online class, and so we have pretty complex analytics and dashboards that show us how students are performing in the class. You can look at a discussion board and you can see that a student forgot to turn it in. But you can also look at this dashboard that will say, “Hey, this student has been late on three of their previous discussion boards. This is a pattern.” So that’s great. AI, some type of algorithmic tool has identified that. We probably could even create an automatic email that lets the student know, “Hey, you didn’t do this thing.” But is that email timed better than mine would be? Is it written in a more personalized manner, does it take that individual student and their entire record of work in the course with me into account? And I feel like again, maybe someday, yes, AI can do that, but right now, it can’t encompass the full range of my experience as an instructor and my experience of that student’s work in the course. I know when a student needs to be challenged, I know when a student needs to be applauded. I don’t know that AI can figure that out based solely on the product that the student either did or didn’t turn in. And so I feel like, again, on one level, this is kind of a personal process for each faculty member to sort of go through their course and think, “Okay, what’s me in this class? What is uniquely me? Where can I bring more of me and my experience to it?” Because really, as much as this was about my students and trying to connect with them, it was really for me. It helped me to reconnect and re-engage with my teaching. Faculty worry that AI is going to take their jobs or do certain tasks better than they can, and I think really the productive way to deal with that is to just keep doing things as humanly, as uniquely, as messily as we normally do them. Let’s say you’re sitting down on a Friday to grade 20 essays. We’ve all been there. We’ve all done that, and we kind of can shift into autopilot at times. Don’t do that anymore, because AI can do that. AI can provide rote feedback and bland email messages. So it’s almost like a wake up call, in a way, for all of us to stop teaching on autopilot and really reinvigorate our practice. I don’t want to say we’re competing against AI, but I do think it’s a moment that’s helping us reassess what’s important in teaching, what’s important in our own disciplines, what’s important in the content of the class, and try to kind of inspire us to get at it, to uncover those things in a different way. So yeah, it’s about reconnecting to that joy in teaching.

Rebecca: One of the things that it sounds like I’m hearing you advocate for is being a bit nimble in how you’re functioning as a teacher, as the technology shifts and change, and as its abilities to do certain things that free up our time to maybe prioritize or put our energy into other things so we can highlight or utilize the technology to do things that are kind of rote in nature, and then places where we can use our human creativity might be where we’d better invest our time. Can you talk a little bit about how we need to shift? For example, you talked about video feedback. There may be a day with new technologies as they’re evolving, like the HeyGen tool, right, where maybe some of that feedback can be generated, and it could be in video.

Nate: Yeah, it’s terrifying. If you stop and think about it for even a minute, this stuff can overwhelm you. So I’m an educator, but I also have a background in marketing, corporate training, more of the business side of things. And I think what you’re talking about, this idea of being nimble in thinking, it’s maybe not necessarily how a lot of academics usually think about things. You learn your subject matter, you get engrossed in it, and then you teach it, and you have that expertise. I feel like maybe it’s just me, or maybe it’s my training, or maybe it’s the different roles I’ve held over the years. I never think I’ve got it figured out. I’m always worried that I’m five steps behind. I’m always trying to think of new ways to do things, not just to make them better, it’s not about necessarily always making things better, but it’s about experimenting and exploring. And I feel like that mindset, that true growth mindset that we talk a lot about as educators, I think that’s part of what we need to apply to this field itself, to this field of teaching. So many faculty members, so many teachers, got into teaching, not to be a teacher, but because they got their masters and PhD in a particular field knowing that they would be a teacher, but without any actual explicit training in it. So again, yeah, it’s this opportunity to stay open, to be humble, to listen, and to explore widely and see what out there might resonate with us in our own practice. So, I mean, Rebecca, you’re asking for particular examples. And again, I guess I might just kind of default and say that those examples are going to be different for so many people. I think there’s a lot of faculty work that can be, and maybe will be, automated. There’s a lot of elements of record keeping, data tracking, things that we do and spend a lot of time on that we maybe don’t need to spend that much time on anymore with AI able to crunch some of these numbers for us or develop data visualizations for us. I mean, I can pull dashboards… I don’t want to scare any of my adjuncts who are listening in my program… but I can pull a dashboard and I can see how often they’re in the class. I can see what they’re doing. I can see their persistence numbers, their retention numbers. I can see all these different things, and I can compare them to other faculty. So I know that Professor X tends to have students who do a really good job understanding course learning outcome 3, but Professor Y is stronger on course learning outcome 4, but not on course learning outcome 3. In order to get all that information, though, I have to look at three different dashboards. I have to go into the class. I have to look at all these different things. And what does that give me? It gives me some element of information that helps me to understand how I might interact with these professors to provide professional development opportunities that might improve their practice in the classroom. Maybe AI can automate all that. Maybe AI can automate some of that data crunching, and then it can just tell me, “Hey, go look in Professor X’s class. Look at week three. See what they’re doing. Use your own human eyes. But I’m telling you where to go look.” We have tools that do this for student work. I think I’m just trying to sort of talk my way through an example of how some degree of faculty work, like in any career, is kind of just a slog. You wake up every morning, you’ve got 30 emails, and it’s just this digital slush that you have to work through to get to the good stuff, interacting with students, working on a new research article you’re trying to tame, maybe these tools can help us get through some of that, while freeing up our time to be more creative, more human, to do those things that we’ve designated are important for us to do as people. And again, maybe that metric shifts for some people. I’m teaching freshman comp. Feedback is really important to me, I don’t want to automate that, but if you’re teaching a 300-level class or 400-level class, and you want to be able to give some students feedback on grammar, maybe that is the kind of thing that AI could help you to provide to a student, freeing you up to deal more with the complex arguments they’re making in their capstone paper. For me, I live in those details, so I need to do that. But maybe that shifts, again, depending on what your role is, where you’re teaching, what you’re teaching, and again, yeah, like you were saying, just to kind of free us up to do things that are more meaningful when we are the one that does them.

John: We want to help our students learn how to use AI ethically and responsibly to prepare them for their lives beyond our classes, but we’re also concerned that they actually learn some basic skills. What are some strategies that you’ve used or you recommend to encourage students to use AI in a productive manner, but not as a substitute for learning.

Nate: I love this because I feel like so much of the discourse that we hear about is how to ban AI usage, or coming up with these really Byzantine ways of getting around AI usage, AI-proofing an assignment, and I feel like that’s going to have its place, because, as we’ve been talking about, eventually AI is going to beat past all of this. So the crux of this is, how do we find ways to use this in the classroom? For me, the key is to normalize and openly discuss AI use with your students, rather than trying to ban it. Students are already using the tools. We know this. There’s an article published every 10 minutes that gives us new data about how we know students are using AI in classes. So we’re not going to get rid of it. Even if your university or your department bans AI usage or limits it in certain ways, students are still going to do it. So by inviting it in, we can try to come up with a way of getting students to understand better ways to use it, rather than just like you were saying, John, rather than just outsourcing and coming up with products where they don’t learn anything, we’ll find ways to integrate it into the classroom. I did some research recently, and kind of basing on a few other ideas I found, I came up with a framework for balanced AI integration in course design and the framework, it’s just four parts. It’s pretty simple, actually. It starts with setting clear guidelines for AI usage, telling students what they can and can’t use, what tools they can and can’t use, where they can and can’t use them in a particular assignment. From there, it means that the assessment itself needs to emphasize the learning process over the product. The third stage is to ensure that you’re encouraging critical thinking, and the fourth stage is to build in feedback and reflection. So what that looks like is that you’re going to talk to your students about AI as a helpful tool for, let’s say, brainstorming or outlining or maybe checking your finished work. It’s not a shortcut, but it’s a way to use the AI in a productive manner, telling them they can use it in a certain prescribed way. Maybe they’ll use it only in that way. But from there, we do need to make sure that we’re modeling that appropriate usage. We’re showing students how we use AI in our own work. We’re talking about it. We’re discussing it. And we continue to position AI as an assistant, not an author, not somebody who is going to generate this material for us, but a way that it can help us get to certain aspects of whatever the assignment might be, emphasizing that process over the product. We’re so used to grading essays. Maybe the essay isn’t what we grade anymore. Maybe we grade the entire process that leads to the essay. It’s not about the thesis statement, it’s about how the student documents their process of developing the thesis statement. Maybe that’s part of how we do this. But I also think we need to make sure that we’re building in reflective tasks where the students can really analyze the AI generated content. They can talk about its limitations, they can talk about the biases. They can talk about how they might have done something different, and analyze what that means, try to figure out what the difference is between human-generated output, writing, versus AI-generated output. Having that reflection built in is something that I think can really help students with this idea of authenticity and authorship. So again, just making sure that you’re explicit in your guidelines, making sure that you’re showing where AI can and can’t be used, and kind of talking about that usage, I think that’s one way that faculty can really bring it into the classroom. It can’t be worse than it already is, right? We’re already dealing with half of our papers being generated fully by AI. I know it sounds almost counterintuitive to say to your students, “Okay, look, you can use AI for this, but you gotta use it in this way,” but I have to believe that if you set up those guardrails, if you work with students, I think they’re going to get to that point where they’re using it as a partner, as a tool, rather than something that’s just wholesale generating their content. I mean, again, this might work differently in a math class than it might in a humanities class, but I do think that there’s generalizability here. There’s a way to make this work in a lot of different environments.

Rebecca: Can you share some examples of how you’ve made it work in your writing courses?

Nate: I can, for example, we’re talking about the idea of generating. One thing I developed is a tool called a thesis generator versus a thesis accelerator, right? So you see these thesis generators online, and what a thesis generator does is a student plugs in a certain amount of information and it just gives them a thesis statement. Again, there’s no learning. The student hasn’t learned anything about what a thesis is, and they certainly didn’t have any real, let’s say, agency or authorship of the thesis statement that’s been developed automatically through one of these generators. And you can go to an AI tool and you can say, “Hey, I’m working on an essay about why I want to be a nurse, and it kind of stems from the fact that one of my parents was really ill when I was a kid, and so I just want a thesis statement that does all that. Make it sound good,” and the AI will do that. So this thesis accelerator tool that I’ve used in a few of my courses is a way for students to talk about their thesis statement without letting the thesis statement be written for them. The AI has been prompted explicitly not to write any material for the student to use in their essay, but to work with them to continue to test their assumptions and ideas, to get them to form this thesis statement. So it asks questions. It asks what the student is interested in, what they want to write about, what they’re thinking about, asking if it really aligns to the topic. It’s asking very probing questions that are pretty explicit in terms of trying to get this thesis developed, but it won’t write it. It will keep working on it that way. But the assignment is not about interacting with the AI. The assignment is not about the thesis statement. The assignment is about the student reflecting on “Okay, I walk into this process. Here’s what I wanted to do. Here’s what the AI wanted me to do. Where does that intersect? What did I come up with? Is that different from what I would have come up with on my own?” and to just really authentically kind of assess, “Okay, this thesis statement that I’ve got, some of these are my words. Some of these aren’t. Some of this idea is mine. Maybe this idea isn’t,” and I feel like that’s one way to get students to understand those limitations, to get students to understand that this is a tool meant to potentially help them, but not supplant them. And I found, as with any assignment, sometimes it works, sometimes it doesn’t. Some students really approach that strongly, and they get a lot out of it. Other students find ways to work around it. But I feel like creating a tool like that, creating a path for students to interact with AI, rather than simply to use it to get information or output. It’s in its infancy. We’re still figuring this out, but I think that’s the way forward to get to this idea of personalized learning that AI seems to be saying it is promising us, this idea that we can have all these personalized learning tools. How do we get there? And I think part of it comes from interaction and conversation. So teaching students in that way, truly dialogic, I think that’s one of the ways we can get there.

John: So one of the things I think you’re suggesting is that faculty have to engage with AI to be able to work with it effectively. And I think throughout academia, there are a lot of faculty who are reluctant to even consider the use of AI and just want to ban its use by students. What sort of strategies would you encourage institutions to do in terms of making faculty more aware of what AI is capable of and how perhaps their students might be using AI tools.

Nate: I think partly it starts with normalizing the use of AI, something we were talking about in terms of students in the classroom with faculty. I think we just need to make sure that there are forums for experimentation. Faculty need to be able to talk about the AI and the ways that they’re using it, they need to be able to share that with their leaders. They need to be able to feel a sense of trust that nobody’s judging them based on things that they’re doing, as long as they’re doing things responsibly, ethically, experimenting. I think that’s how we learn. And having forums for that, whether it’s a community of practice, whether it’s discussion circles, whether it’s a Friday open office hour, some way to share and talk about the gains, the wins, the losses. I think that’s a first step of taking an organization, an institution, a university, and helping it kind of take those steps toward AI use in higher ed. But I think that an institution can say, Okay, we’ve got this really forward thinking process with AI, and we think it’s okay to use it in X, Y, and Z ways, and we’d like you to experiment and explore. There are still faculty who won’t do that. A lot of faculty just aren’t going to want to learn this. And so [LAUGHTER] I’m not entirely sure the answer to that question, I think that’s the underlying problem with so much of the work that’s being done with AI and higher education, is the fact that you need to have willing partners, you need to have supportive institutional governance. You need to have supportive infrastructure in place, but you also need faculty to truly engage with it with an open mind. And I’m not saying that that’s bad or good. Everyone’s allowed to do what they want to do. I know a lot of faculty will never want to use this. The sort of pervasive idea that we see with AI is that we should ban it, it shouldn’t be used in the classroom at all. So many of my colleagues fall on that side of the spectrum, and that’s fine, but then I guess it’s a process that’s going to take time. I can only imagine what it was like for teachers when a principal walked into the office, the bullpen office one day and said, “Hey, you know what, tomorrow for your quizzes and tests, I want you to use the Scantron form. It’s weird. It’s different. It’s totally new. I want you to try this Scantron out,” or the day somebody first put an overhead projector on a teacher’s desk and was like, “Use this to teach. Give me your chalk. Start using this thing.” I feel like it’s fundamentally changing how we can interact in the classroom with our students, with each other. And if you don’t want to engage that, think about that. I guess that’s your prerogative. Famous story in my family, my dad retired from his job the day they put a computer on his desk. He walked out the door that day. I think it’s going to be a slow process, but I think eventually we’re going to get to a point where faculty are going to be able to see the benefits of this. We’re going to show the benefits to them. We’re going to have to run workshops. We’re going to have to run dedicated professional development tracks that explain the small wins and the large wins of using AI, some of the upskilling that we were talking about earlier in the conversation. If we show faculty this isn’t just about generating content, but it’s about learning while we’re doing it. It’s a tool for reflection, an aid for reflection. I think the more we can develop these use cases that show that and we can share those, I think the more we’re going to win people over. We’re not going to win everybody over, and I think that’s fine. There are probably a lot of people who still won’t use Scantrons, but I feel like it’s one of those moments where getting to the hearts and minds of faculty is what’s really crucial here, and a lot of the conversation around this doesn’t always seem to be productive for that. We’re talking about limiting AI or banning AI. I think we need to have more conversations like what we’re having today: ways to productively include AI, and then let’s talk about them, let’s delineate them, let’s share them, and let’s see if it resonates for some people. Maybe it will for others, maybe it will for some, and then it sort of spreads.

John: And we should note that in the podcast released a week before this one, we addressed some of those issues in terms of what’s being done at several campuses in SUNY, and we’ll include a link to that in the show notes. But one of the things we’ve noticed here is that many of the faculty were really resistant to the use of AI. Once they were in a workshop and got to see how they could use AI to help improve their teaching, they became much more aware of possibilities, and their attitude changed really dramatically within even just a few hours of that professional development. But the key is getting people to that professional development work, and it’s sometimes hard to get people started with that, especially if they have this fear or this serious concern about ethical issues associated with AI.

Nate: I think that’s why finding ways to show how AI could help do work you might already be doing is one of the ways in for faculty, especially. I mean, what you’re talking about is the same kind of thing we see in studies. People don’t want to use AI once they do it, because they see how it works, they start to understand, okay, this can be a powerful tool. They may still sort of ethically make a choice not to use it, I suppose, but I feel like it’s just seeing how it works. Can be the gateway for so many people. So what’s that gateway for faculty, but to show them how it might help influence their work, how it might help free up time to do the creative things we were talking about earlier. I feel like the more we can give examples of that, I think the more we can lead people to it. And it means starting small. It means looking at what we’re already doing and seeing if there are ways this can be made to help assist in doing it a little bit better.

Rebecca: Along the same lines, one of the things that I’ve observed in the way that institutions are handling AI is typically, right now it’s voluntary, like professional development around AI is voluntary. And so you’re getting folks that are at least curious or completely resistant because they want to fight against it, so they’re showing up to things, like there’s some motivation to show up, and it’s their motivation to show up. But if we start meeting faculty where they’re already at in department meetings and other places that are places where they’re already kind of required to be to at least raise the issues or the conversations that’ll continue involving some additional people.

Nate: That’s the hope. One of the things we’re going to start piloting this with the new academic year is we have a roster of course revisions that are going to happen over the course of the year, and we’re going to sit down with faculty experts, and we’re going to say, “Okay, you can follow the old 12-week model of revising your course, where you’re going to meet with people, you’re going to develop the content, or you can follow this new model. It’s nine weeks long. It’s a little shorter. It’s going to use AI to help assist you. Which way do you want to try and if you pick the nine week way, we’re going to document that, we’re going to share that out as a kind of story, good and bad, whatever happens.” And I feel like that’s Rebecca kind of at the heart of what you’re talking about, this idea of finding ways to get faculty to connect with it that are kind of low lift. You’re going to do this thing anyway. Do you want to try this other possibly easier, possibly faster way of doing things, just to see what happens? You want to help us figure this out? And I think that might help. I love being an academic. I love teaching university. I love my students, but you read these emails from Microsoft or Asana, or I mentioned Ethan Mollick earlier. He has a substack, and you just see the massive amounts of money that businesses are pouring into AI training. They’re mandating it. They’re throwing everyone into day-long workshops. They’re testing and piloting. They’re opening up tools and universities and institutions just aren’t doing that. We’re not as well funded, and maybe we’re a little stodgy, maybe we like to do things at our own pace. And I respect that and appreciate that, and that’s one of the reasons why I’m an academic and not a CEO, one of the many reasons I’m not a CEO. But I feel like sometimes I read those things and I just think, “Boy, how nice would it be if we could just mandate everybody an AI exploration day,” you know, just play with a new tool and see what you come up with. Because professional development, educational development in higher ed, is so different from what I experienced as a corporate trainer, where people were thrilled. They might make fun of it. They might think the outcomes weren’t going to be worth it, but they were there and they were active and they were engaged, and as we’re intimating here, it’s tough to get people to show up, it’s tough to get them to engage, it’s tough to track if they learned anything, and it’s tough to check in six months later and see if they’re still doing it. So what does that look like? That’s the challenge of educational development overall, probably. But maybe it’s small wins. Maybe it’s small communities that catch fire. I don’t know.

Rebecca: I wanna underscore one word that you said, which was play. I think that’s something that we don’t often allocate time to play around with new tools and technology, and we’re putting our time resources towards something else, but if we put time resources towards play and experimentation, we might find something that’s useful.

Nate: I think we just have to engage with that growth mindset. We have to be like our students. We have to be learners. We have to listen. We have to just explore, instead of always jumping in with the information, with the answer. We have to sit back and maybe that play comes from it. One quick example, my colleagues are probably not listening, so I’ll share this. I woke up one day and I had an email that had kicked off the night before. There were like six or seven of us on the email, and it was probably at that point, like, 16 or 17 emails long. I clock off, I go to bed, wake up the next morning, and there it is, staring at me. And I was like, boy, this is exactly what people talk about that, “Hey, I can help you with your email.” So I grabbed all of that, put it into a protected AI tool that’s not going to train itself on my data. And I just said, “Is my name mentioned here? Are they talking about comp? Like, why am I involved in this?” And it spit back some really good, useful information. And I was like, “Okay, great.” And then I had 30 minutes to play, to just do something else, something different, and instead of engaging in the back and forth politics of a gigantic email chain, I got to cut through the noise and do something that was more meaningful for me in that moment, and that was that play of exploration, trying something new. If we can get back to that and can enkindle that, I think we’re going to get there.

John: I did enable the Apple intelligence on my phone, which is only a few months old, and it does give me email summaries automatically. They’re often somewhat distorted. In fact, they’re often so scary it forces me to look at my email sooner than I would have otherwise, [LAUGHTER] because it does look like there’s something serious that I have to address right away. But it’s getting better. Those types of summaries can be really useful.

Nate: I remember when I first turned on AI summary on Zoom, we do a lot of work on Zoom as an online institution. I first turned on an AI summary, I was in a meeting, and afterwards, I’m looking at the summary, and I’m like, these aren’t the things we talked about at all. You got everyone’s names wrong, and this is really bizarre. And not even three months later, I was looking at it again, and I was like, this is actually now really good. And so along with playing, I think it’s that sense of continuing to grow and to learn to keep coming back, don’t play with it and then write it off, but continue to explore, continue to play, and to also take what’s there to be taken and don’t over rely on it. We don’t want our students to just over rely on this. You don’t want me to never read my emails again, just reading summaries of emails. But I do think that there are ways we can develop that mindset, really a deep sense of AI literacy as an important outcome in and of itself, where we realize how to interact with these tools in a way that’s going to help us free up that time. Do better work more as opposed to the slush work more. And we get buried in our emails. We get buried in grading. We get buried in so many different things. There are ways to shortcut some of that while still doing the meaningful stuff. I’d like to figure out what that is.

Rebecca: Sounds like a great note to get to our last question, which is we always wrap up by asking: What’s next?

Nate: Well, what’s next for me is probably grading papers. It’s… [LAUGHTER]

Rebecca: Wait. No, that’s the slush. [LAUGHTER]

Nate: For me, that’s important work. We’ve been talking about this a little bit. For me, I think one of the things that’s really important is to continue to learn alongside other people. That is how I’ve learned the most. I’m a teacher, but I’m also a teacher of teachers, and one of the ways I learn how to teach my students better is by working with them. One of the ways I learn how to interact with my other faculty members, the faculty members that I oversee, is by being a teacher myself, seeing how they do things, working with them, helping to figure out what those elements are that can be honed or refined or developed. And so to me, it’s always just about learning. It’s about putting myself out there, trying to figure out what can be figured out, and being curious, listening, finding ways to think strategically, to take the small thing I might be doing and think about it in terms of how that might scale to other classes, other teachers. I’m playing around with AI on my computer in the morning. I’m trying to figure out different ways of doing things. While I’m doing that, I’m also thinking, “ooh, how can I convey this to somebody else? How can I best show this, demonstrate this to help other people get excited about it?” So it’s about structured play. It’s about strategic thinking. It’s about having fun and getting back to the root of what brought me to this crazy discipline to begin with. I want to ask better questions. I don’t really want to know the answers to things, because that always seems like the end. I just want to keep asking questions and getting to new places along the way.

John: Well, thank you. This has been a fascinating discussion, and it’s one that is really important for people to be having right now on every campus.

Nate: Yeah, I appreciate it. It’s great to talk about this stuff. And like I said, you guys kept me on my toes a little bit here. It’s fun to think out loud, and I appreciate your time today, John, Rebecca.

Rebecca: Sounds like a really human activity that just occurred [LAUGHTER] that has maybe a relational aspect, right, that maybe can’t be replicated by AI.

Nate: I mean, as podcasters, you guys have explored with the Google podcasts, right?

John: Notebook LM? Yeah.

Nate: …where you can give it a research article, and you know you’re listening to it, and it sounds like a podcast. It sounds like you’re listening to NPR, but the information is wildly wrong, and it’s missing that nuance and these elements of redirecting things, stuff that’s coming up that’s not scripted. Some of the things we talked about today were not questions you shared with me that we might talk about, and that’s great. Yeah, that’s the human enterprise. It’s messy, it’s weird. Can’t be predicted. That’s what makes it fun and worthwhile.

John: Although notebook LM is getting better, as all these other tools are as well.

Rebecca: More deep fakes ahead. [LAUGHTER]

Rebecca: Thanks for joining us.

John: Yes, thank you.

Nate: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

398. Multicampus AI Initiative

Faculty are faced with the need to adjust instructional strategies in response to AI. In this episode,  Racheal Fest and Stephanie Pritchard join us to discuss a professional development initiative for faculty involving six campuses.

Racheal is a Pedagogy Specialist at the Faculty Center for Teaching, Learning, and Scholarship at the State University of New York at Oneonta. She also teaches writing courses in the English Department. Stephanie is the Coordinator of the Writing Center, the Coordinator of Writing and Ethical Practice, and an instructor for classes in poetry and English composition here at SUNY Oswego. Racheal is the Principal Investigator and Stephanie is one of the campus coordinators on a SUNY multi-campus grant focused on faculty development related to AI.

Show Notes

Transcript

John: Faculty are faced with the need to adjust instructional strategies in response to AI. In this episode, we discuss a professional development initiative for faculty involving six campuses.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Racheal Fest and Stephanie Pritchard. Racheal is a Pedagogy Specialist at the Faculty Center for Teaching, Learning, and Scholarship at the State University of New York at Oneonta. She also teaches writing courses in the English Department. Stephanie is the Coordinator of the Writing Center, the Coordinator of Writing and Ethical Practice, and an instructor for classes in poetry and English composition here at SUNY Oswego. Racheal is the Principal Investigator and Stephanie is one of the campus coordinators on a SUNY multi-campus grant focused on faculty development related to AI. Welcome Racheal and welcome back Stephanie. It’s been a while.

Racheal: Thanks for having us.

Stephanie: We’re very happy to be here.

Rebecca: Today’s teas are:… Racheal, are you drinking any tea today?

Racheal: Yes, I actually came prepared with two teas. I’m a big tea drinker, and I’m starting with an organic loose leaf orange pekoe, and then that will be my allotment of caffeine for the day. And then I’ll be drinking an herbal tea, which is a honey turmeric chai.

Rebecca: Both of those sound delightful, but maybe you’re slightly an overachiever in the tea category. How about you, Stephanie?

Stephanie: I am drinking my go to Earl Grey tea, and then perhaps this afternoon, will indulge in some blueberry green tea.

Rebecca: I knew that Earl Grey was gonna be involved.

John: We do have, by the way, I think, four different varieties of Earl Grey tea at the teaching center, and I think Stephanie has tried each of those at one point or another.

Rebecca: How about you, John?

John: I have a spring cherry green tea. Now that it really is spring here, we haven’t had snow for over a week.

Rebecca: At least today, it feels like spring. We’ll see what the dice rolls tomorrow. I have Chai today.

John: We’ve invited you here to discuss this multi-campus grant funded AI professional development program that you’ve been working on all year. Can you provide an overview of the program?

Racheal: So, this program is called Teaching with AI: A Cross-Campus Community of Practice, and essentially we brought together faculty from six SUNY campuses, including regional comprehensives, technology schools, and community colleges, and we asked them to design learning activities that integrate AI into the classroom in ways that are critical, active, and inclusive.

Rebecca: So you mentioned that there’s six campuses involved. Can you talk about how many faculty are participating?

Racheal: We have about 66 faculty participants at this time, and the majority of them are funded by our IITG funding. But we were able to get some additional funding at Oneonta and Oswego as well. And Oswego, as Stephanie and John, you can attest, had a really big and enthusiastic cohort.

Rebecca: For those that aren’t familiar, can you describe what an IITG grant is?

Racheal: The IITG grants are innovative instruction technology grants. They’re awarded across the SUNY system, and they’re peer reviewed through a competitive process that is used to distribute funds, especially focused on educational technology, and often OER (open educational resources).

Stephanie: We are participating with six other SUNY campuses, and those campuses are Alfred, Morrisville, Oneonta, Orange, Oswego, which is my campus, and Schenectady.

John: Could you talk a little bit about what the participants in this program have been asked to do as part of their work in the program.

Stephanie: Our project began in January, and we started with a webinar from Anna Mills, who is an advocate for AI literacy and also for OER or open educational resources. She teaches writing at the College of Marin, and we made her webinar available to all of the participants in our grant. But we also had some interest from SUNY, so we advertised the webinar through SUNY’s Center for Professional Development, and we had a very large group of participants attend Anna Mills’s talk, which really focused on thinking about AI, how it’s affecting the different assignments that we’re giving in our courses in all sorts of different ways in higher education. She also spoke about some strategies that teachers should consider as they are really working to either adapt AI into their courses or sort of try to manage how AI is affecting the classes that they teach.

Racheal: And I’ll just add that one thing I really love about Anna is she is focused on critical AI literacy, and that was a big piece of our project, as we’ll talk about as we develop the details, but she has been a voice that is kind of guiding conversations about evaluative engagement with AI and not only ways to integrate and use it in the classroom, but also ways to think about how to protect learning objectives now that these tools are widely available.

Stephanie: So we started our project by listening to Anna Mills, like I mentioned. It was very well attended, and that was in early January. After that, each of our campuses had a day-long kickoff event where participants in each of our cohorts gathered in person on the various campuses, and we led them through a series of structured activities and introduced them to the grant objectives and deliverables. We also shared with them the Brightspace shell that we had created specifically to help guide our cohorts through this process.

John: And I’ll add that in the initial meetings, one of the things that was really helpful in getting everyone there, besides requiring people to be there as part of the process, there was also lunch available, which made it a little bit nicer, and it created a very nice environment for people to begin their work together.

Stephanie: It was a very nice way to pull everyone together on our different campuses, introduce them to one another, and also sort of see what level of knowledge or engagement our different participants had at the very beginning of the project. We had participants on our campus who were doing all sorts of different things. Some had already been integrating AI into their courses in various ways. Others were experimenting with ChatGPT for the very first time during that day-long kickoff event. So it was really interesting and kind of energetic and wonderful to see all of these different people in a room learn what they were doing and also learn about what they wanted to do and what goals they wanted to achieve with the project.

John: And I’ll also note that some of the faculty were there mostly to learn how to keep students from using it as an alternative to learning. Once they started seeing, over the course of that day, ways in which they could use it and perhaps have their students use it productively, t here was a pretty significant shift in their attitude towards AI use over just the course of that very first day, which has continued over the project.

Racheal: That’s a great point, John, we saw that here at Oneonta as well, and I think that we have a number of faculty on our campus who are quite enthusiastic about AI and have been adopting it, and some of the other diversity that we saw in the room that I thought was really interesting. There were folks who were more skeptical about AI or more invested in kind of protecting learning objectives, and I did want to make sure that the experience helped them to think about those problems and questions, because those are so pervasive. And then there were faculty who also have been adopting really enthusiastically in different ways, so creating a space for both of those groups to kind of learn together and hash out different positions, even while maintaining those different positions, because I think AI’s integration, it really varies across disciplines, and I think that that richness and diversity was a really valuable and important part of this project.

Rebecca: So beyond that first kickoff day, what were some of the other things that faculty have been required to do as part of the project?

Racheal: So after the kickoff, we moved on to a series of monthly meetings that we had planned, and those meetings were designed to continue the conversation and also to offer pedagogy support around designing learning activities. So in addition to thinking about AI integration, we were also offering that pedagogy support of thinking about backward design, thinking about how to begin with course objectives, how to design an activity that is active and inclusive for students, and then how to think about feedback and assessment. So we’ll talk a little bit about the rubric that we use. I mean, we did use it to assess faculty deliverables, but it was more of a tool to guide the conversations about what a strong learning activity could look like in a range of contexts. So that rubric kind of guided our monthly meetings. We were focusing on having an AI policy in your syllabus, developing at least one course objective related to AI. And then we were focusing on designing the learning activity and finally offering feedback and assessment on the learning activity. And we’re wrapping up those monthly meetings in April. Some campuses have already hosted them. Mine is actually this Friday coming up, and at that April meeting, we were giving folks a chance to share and give each other feedback on drafts of their assignments and learning activities before they submit those final drafts.

Stephanie: One of the things that we tried really hard to do in those monthly meetings was to model effective ways of teaching to everyone who was participating in our program. So in addition to the things that Racheal mentioned, like backwards design, we also spend a good chunk of time focusing on the TILT approach. For example, they had an option, they could do an in class learning activity with AI or they could have their students submit something in an out of class way. So we spent some time talking about what those assignments could look like, and why that level of transparency in your expectations as a teacher matters, and why that’s really important. We also spent time, as Racheal said, talking about feedback and all of the different ways that we give feedback to our students, whether it just be a grade or written feedback or verbal feedback, using a rubric or not, using peer feedback as a successful strategy. So we really tried, during these monthly meetings and through our Brightspace shell, to really model what effective teaching looks like, in hopes that our participants would take that and then apply it to the projects that they were developing.

Racheal: And another important piece that pairs with that, I think, Stephanie, we also tried to be really responsive to the different needs and contexts that our faculty were bringing. If you were planning an in class learning activity, for example, you’re not going to grade that or provide written feedback, and we certainly weren’t expecting that. We wanted to allow for a range of approaches, so we talked about strategies for giving feedback during class or through conversation. We tried to ensure that a range of different approaches to learning activities and assignments were represented in the conversation.

John: And just following up a little bit on that, one of the reasons we focused on the TILT framework is because, in the current environment, students may choose to use AI as an alternative to learning material, and we’d like them to be aware of why we’re asking them to do specific things, because if they understand that they’re developing skills or tools that are going to be useful for them later, it’s more likely that they’ll actually engage in it actively themselves, rather than trying to skip the actual learning by using AI. So we’ve encouraged faculty not only to do it in these activities, but also to integrate the TILT approach into all their teaching and learning activities.

Rebecca: For those that aren’t familiar, TILT stands for Transparency in Learning and Teaching. We do have a previous episode on that, that we’ll put in the show notes.

John: And we’ll share a link both to that and to Mary-Ann Winklemes’s website, which describes this as well. And just as one more aside related to this, if you’re not sure of how to create a TILT assignment, you could always feed an assignment into AI and ask it to put it into a TILT framework. And that works especially well if you also share your course learning objectives with the tool as well.

Rebecca: One of the things that I’ve heard all three of you mention is this Brightspace course that you set up for faculty. Can you talk a little bit about this?

Stephanie: So we spent some time as a leadership team developing the Brightspace shell. We knew that we wanted a common place for all of these materials to live, especially the support materials, as our participants were working their way through our different months, or our different modules. So we worked with members of the leadership team to develop this Brightspace shell that could kind of serve as a common place for us all to come together. In that shell, in addition to uploading our monthly materials that would focus on a different piece as we moved through that rubric that Racheal was talking about earlier, we also established clear guidelines for all of our participants so that they could understand what their requirements were as they moved through the program, which we’ll talk more about in a little bit. But that Brightspace shell also offered opportunities for our participants to talk to one another. One of the things that we required our participants to do was engage with one another in what we called our community of practice discussions. So our participants were required to engage with a small group of their colleagues from different campuses. This was a little bit interesting, because Oswego and Oneonta had larger cohorts than some of our other participating campuses. At Oswego, we were lucky enough to receive some additional funding and support from our Chief Technology Officer, Sean Moriarty, which we really appreciated. And I know that Oneonta had some extra funding as well. Is that right, Rachael?

Racheal: That’s right. We had initially requested funding for 10 faculty to receive stipends of $600 each, but our Faculty Center for Teaching, Learning, and Scholarship was able to provide additional stipends for, I think for us, it was six more members to participate in the program. And I think for you, you were able to double that number.

Stephanie: So, Oswego ended up with 24 participants total, which was about 14 more than what we had initially planned for. So in the community of practice discussions, it was a little bit heavier with membership from Oswego and Oneonta, but we were still able to design these discussions based off of what our participants requested. So some wanted the opportunity to speak with people from other campuses who were in similar disciplines as themselves, and some wanted to be in groups with people who were from totally different disciplines to sort of learn what other people were doing.

Racheal: I think we’re bringing out here the two tracks that ran parallel throughout the course of this experience. On the one hand, we had our in-person cohort meetings on each of the six campuses, and that allowed people to connect across departments from a range of disciplines with folks on their own campus. And then we had this second element that Stephanie has been talking about, that was the cross-campus community of practice element, and those were virtual meetings with three or four faculty members per group. And we asked those cross-campus groups to meet for a minimum of two times and then to post their takeaways to the discussion area in Brightspace. That way we could track and see what our faculty were talking about. We could take a look and answer questions that might have come up. And we also, as a leadership team, responded to those discussions on Brightspace, and we really saw the groups talking about a range of topics. One of the reasons that we wanted to connect faculty across campuses and not just run these cohorts on individual campuses, is that we’re seeing faculty bring a lot of different approaches and attitudes to AI, and a lot of that is place specific. A lot of it is related to institutional cultures or resources, and so the ability for faculty to meet from across the system and to share what their institutions are doing to support professional development related to AI, as well as to share what different departments or what conversations around AI might be happening in those different spaces. We wanted to create space for people to come together and have those conversations.

Stephanie: It was also really interesting, because participants in the cross-campus groups, in addition to having these other conversations about AI and pedagogy, began to really work well together as a unit as they worked on the deliverables for this project. So they worked together to share their ideas for their learning activity, their learning objective, their syllabus statement. And then as we got deeper into the project, they started to share drafts with one another and to ask each other, is this meeting the criteria for the grant? Are we making a learning activity that’s critical and active and inclusive? Do you have feedback for me? So as we talked about earlier, we had the meetings on each of our individual campuses with our campus groups, but it was really nice that the cross-campus groups were engaging with one another beyond what we had initially imagined, which I thought was really a nice bonus from this.

Racheal: I completely agree, and to try to facilitate some of that conversation as a leadership team, we didn’t want to impose structure on these meetings, but we also wanted to provide some resources for getting those conversations started. So we created on Brightspace question banks that faculty could choose to draw from to guide those conversations. And so with those suggested structures, I think we were able to leave room for that creative exchange. And each group was able to kind of determine how they wanted to use that cross-campus group time.

Stephanie: We also had, in addition to the cross-campus group discussion boards, we also made additional discussion boards that were optional, so participants could choose whether or not they wanted to participate in a board called community conversation, which was very informal, just a way for people to share their thoughts or ask questions to one another. And again, this was not required, but we did have participants posting there, engaging with one another in that way. Then we also had another discussion board that we called news and tools, which was basically the opportunity for participants to share new tools that they had stumbled across, or post things related to things that were happening as new AI tools were developed and came out. As we know they’re changing so rapidly. Things are coming out every single day, so participants had the opportunity to share that information with each other as well, and I think it was nice to give them the opportunity to participate in more than just one forum, because, as Racheal said, we had several members who were very engaged in the project, and I think valued other spaces to collaborate.

John: What are some of the examples of learning activities under development by faculty?

Stephanie: So, materials for this project are not due for another two weeks or so, but we have had several participants submit their materials already, getting things all squared away before the madness of final exams week here. So we did have a few faculty who were thinking about engaging with AI tools as part of the brainstorming process. So for example, one of our participants here gave an assignment where students were asked to use Perplexity, which is a free AI tool that also can access the web and provide suggested sources. He’s asking his students to use Perplexity to help them continue brainstorming for an assignment that they’re working on in the class. So students already have some general ideas of what they want to do, and he’s asking them to engage with Perplexity to deepen their brainstorming. So for example, ask Perplexity to provide some suggested sources to the topic that they’re considering, and then evaluate the sources that Perplexity is suggesting. He also asks them to prompt Perplexity, to give them a counter argument to what they’re exploring. So those are some examples. T hat would be an example of a learning activity. It’s something that he’s going to have them do in class. And the nice thing about his prompt is that he gave his students four different sample prompts that they could take and then put into the tool, but they would have to fill out the details of their particular brainstorming. So that’s one example from our campus. Racheal, would you like to give one example from yours, and then I can do another one?

Racheal: Absolutely, that sounds great. I’ll share an example from Oneonta, while saying as well that the leadership team will be taking a look at the examples across campus once folks are uploading them, as Stephanie has said, but we haven’t done that yet, so we only have the knowledge of folks we’ve been working with on our own campuses. So one faculty member in sociology here has been developing a project that I think is really interesting and exciting. He’s going to be teaching a course in the fall that has a unit focused on conspiracy theories and thinking specifically about how to debunk or counter conspiracy theories. And so he has identified an AI chatbot called DebunkBot, which has been developed, I think, out of Cornell. And he’s going to ask students to identify a conspiracy theory of their choice, engage with DebunkBot to see how the AI counters that conspiracy theory with factual information, and then they’re going to evaluate those AI outputs. And I think within the conversations he’s having with students, too, is the question of, do facts work against conspiracy theories? There are various opinions on that, so not only evaluating DebunkBot’s outputs, but also thinking about it as a tool to counter conspiracy theories. What are its strengths and weaknesses? And what I really like about this project is, I think that it exemplifies that critical element that we’ve been emphasizing through this project. By critical, we really mean evaluative. We are asking that all of these learning activities and assignments take up that AI literacy task of helping students evaluate AI outputs and evaluate AI as software.

Stephanie: Tagging on to the evaluative piece, one of our colleagues here is working on an assignment that asks students to sort of reverse write a paper. So she’s giving her students a prompt and asking an AI tool of their choice, probably ChatGPT I imagine… one of the components of our grant that we really emphasized was trying to make these assignments or learning activities as inclusive as possible. So we’re thinking about access, who has access to this technology. And since ChatGPT doesn’t require users to make an account, there is a free version, a lot of faculty are leaning toward those sorts of tools for this. So anyway, one of our faculty here is asking her students to have ChatGPT generate an essay on one of the books that they’re reading for the course, and students are then going to look at the essay and evaluate the output that ChatGPT has created. So they’re going to look at things like whether or not the material is accurate, if there are hallucinations, if it is in fact doing what it was asked to do. So I think they’re being asked to analyze, “So how is the quality of ChatGPT analysis?” And this faculty member also has a series of follow up prompting questions to have the students continue to engage with this essay that was generated and see if they can make it better, or see if that’s it, if it’s as good as it’s going to get, and if there are still holes in the analysis or with the argument. As a writing teacher, I thought that that was a really cool way to look at what this technology can do, really consider its limitations as well, and think about how are we going to do writing, teach writing as we keep going, as more students rely on these sorts of tools.

Racheal: I love that you were emphasizing the inclusivity piece there, Stephanie, which has come up in our conversation where you were mentioning especially how you emphasize TILT on Oswego’s campus. On Oneonta’s campus, we talked a little bit about UDL, universal design for learning, but those were frameworks that we wanted to keep in the mix as part of the conversation, we didn’t necessarily have time in our progression to really dive deeper focus. So we set some basic guidelines for inclusive activities that we encouraged everyone to meet. And those were, as you said, making sure that whatever AI tools you’re asking students to use they have equal access to. So that would look different on different campuses as well. Here at Oneonta, we’re a Microsoft campus, and that means that we have Co-Pilot for all students over 18. And if you’re thinking about, when you’re introducing students to AI tools, not only making sure that everyone can access it, but also making sure that you’re thinking and talking about who might have access to paid versions, because students are sometimes subscribing to ChatGPT at that higher level, and just making those conversations about what tools you’re using part of the conversation. The other inclusive piece that we really emphasized was just that providing instructions to make sure that you’re aware that people… students, specifically… are bringing different levels of knowledge of technology to the classroom. Some students are using AI all the time, others have never used it and don’t want to. I think we’ve had those conversations in our group. So just being aware of the diversity of student capabilities as part of that inclusivity and accessibility piece was really important to us too.

Stephanie: Just to kind of tag on, I thought it was really interesting that, as we talk about access and who has access to a paid tool versus a free tool, I’m sure you have seen, I know John and I have spoken about this a little bit that, right now ChatGPT has a free trial of their paid version that they’re marketing specifically to students. So it’s ongoing from now until the end of May, which I thought was really interesting. And I know that they’re not the only tool that has done that. Is that, right, John?

John: Gemini and Claude have also done it, and one of them I forgot, which one actually is extending it until next year. So there’s a little bit of competition going on to get more students using their particular models, because student use does make up a remarkably large share of the use of all the AI tools now, for reasons that perhaps are not always as positive as we’d like.

Stephanie: I think it will be really interesting to see how that continues to develop as we move forward. They’re doing this two month free trial. Now. What does that mean for the beginning of the fall semester and beyond that?

John: One thing we should also note is that this podcast is coming out in June. We’re recording this in late April, which is why snow was as recent as last week, and why many of the references are towards the last stages of the semester. This is a rare occasion for us to be this far ahead in our recording. We often record things a week or two before their released, but we’ve happened to be able to get a lot of podcasts scheduled within a fairly short period of time. Along the lines of the assignment that Racheal mentioned in evaluating sources, Mike Caulfield has been developing a prompt for SIFT, which is his approach for analyzing the veracity of online claims, and as of last night, he’s developed an expanded version of that prompt, and we’ll share a link to that in the show notes as well, because we can use AI tools to try, not only to create false information, we can also use it to try to verify claims.

Rebecca: One of the things I’ve heard you all talk about quite a bit, and I’d like to hear a little bit more about, is the rubric that you were using to evaluate the assignments and things that faculty were creating.

Racheal: Yeah, the rubric that we developed, as I mentioned earlier, we really thought of this more as a tool to guide our conversations, rather than as an assessment piece. But we’ve run at SUNY Oneonta, similar cohorts devoted to different learning activities and aims in the past, and we knew from those experiences that being really clear about what the deliverables are and what elements we’re privileging and would like to see, that’s going to help faculty design learning activities that then can move as we’d like to move them, to our public facing repository. So we focused on three categories of evaluation on the rubric. First, we were asking folks to develop an AI policy for their syllabus. That was a big part of this project, too. As we are thinking about helping faculty and students navigate the emergence of AI, students are coming into a range of classes that all have different expectations for them. So some faculty might be saying, “if you use AI at all, you are cheating.” Others might say, “I want you to use AI. You have to use it. And here are some of the ways that I’ll be evaluating your use of it.” So just thinking about how we as a leadership team could support students by encouraging AI policies through this experience was important to us, and those policies would capture a wide variety of approaches to AI in the classroom. Some policies that we’ve seen have given guidelines assignment to assignment, where some assignments require AI use, some ask students not to use it. We’ve seen policies that outline the specific ways students might use AI throughout the semester, and we’ve also seen policies that said you can’t use AI except in these very specific ways that I ask you to on this learning activity. So we’ve really seen a range there. And the rubric begins with that AI policy element and with the course objective. So one of the elements that we’re collecting as part of the deliverable with the learning activity is the syllabus with an AI policy and a learning objective.

Stephanie: And I think it’s important to emphasize as well that regardless of how faculty feel about AI, like whether they’re embracing it or they’re saying, “I really would prefer you not use AI in this course.” It’s really important for us to be transparent with our students about what the rules are. As Racheal said, our students are generally taking five different courses or more with five different faculty who have a whole bunch of different feelings about this. So some people might think I don’t want them to use AI in my course, so I’m just going to not put anything on my syllabus. It’s so important for our students that we are transparent, regardless of how we feel about it.

Racheal: Yes, and that was one of our aims with including that syllabus element. So the second rubric element focused on the learning activity itself. And again, that had the critical, active, and inclusive piece. I think we’ve done a good job talking about how we define critical as evaluative, and how we define inclusive as tool access and instructions. And I guess I can just briefly define active as students are using the tools themselves. So we’re not just asking you to demo AI in a lecture. We’re saying, “Get students in there evaluating outputs for that AI literacy piece.” And I do think it’s important to mention here that we had many conversations about students who want to opt out of AI tool use, students who might have environmental reasons for not wanting to use the tools or other ethical reasons, and we did talk about strategies for that, including alternatives that would allow students to maybe engage with an output that the instructor created from AI or alternative assignments that aren’t actually asking them to engage with AI, but might be asking them to evaluate AI in a broader conceptual way. So those alternatives did come up for us. And Stephanie, maybe you could talk about that last rubric piece, the assessment and feedback element.

Stephanie: Sure. So we thought that it was important for our faculty, as they are designing this assignment, to really think about how this assignment or learning activity would be assessed. How are they going to give feedback to their students as they experiment with these tools, as they evaluate these tools, and what does that assessment piece mean? So we have some participants who are just doing one assignment or one learning activity, so they have to just think about that one piece. But we’ve had several participants in this project who have become inspired by the one part and have decided to make lots of changes to their assignments or their learning activities, and then think about what assessment means for them and for their students. So as we talked about earlier, the feedback piece is very important, as our students are critically and actively engaging with these tools, because we have to guide them through what they are learning. If we are really focusing on AI literacy and helping our students understand or begin to understand how these tools might impact, not only the work they’re doing in the classroom, but their future jobs. Every career is going to look a little bit different. Some are really going to be embracing these tools and expecting our students to understand how to use them, where others perhaps not so much. An example, I teach creative writing courses. Right now. In those courses, I am not emphasizing AI use because, for many of my students, they’re looking to submit their work to different journals, and most creative journals at this time are not accepting AI-generated work. So it’s important to acknowledge that, and it’s also important to focus on why we’re going to continue to do things the way that we’ve done them a little bit for the past, right? But other courses, if I’m teaching English composition, and I’m teaching them different ways that they can outline or draft or brainstorm, how they can use AI to get feedback on their own work and their own writing. Those are important skills that will come in handy for them later on, as we talk about things like efficiency and making more polished drafts and getting feedback before you turn something in. Those skills will come in handy more so for those students than for some of the creative writers.

John: We have a student advisory board for our teaching center, and they met last week, and one of the things that came up was the topic of AI, and two concerns that students discussed pretty actively was that a lot of faculty have not made clear yet what their expectations are about whether AI use is allowed, when it is allowed, and so forth, and they noted that it was more rare than the norm for people to actually have made those policies. We’ve obviously been encouraging faculty to do that for a couple of years now, but it hasn’t made it out to all of our classes yet. But our institution will be requiring an AI policy statement in the syllabi beginning next fall. So this transparency issue and being clear about your expectations is really important. The other thing they were concerned about is being falsely accused of AI use, and this is something that is very concerning for students, and we have a lot of cases where some faculty are just completely ignoring AI, and others are assuming that AI is being used by all students all the time, and accusing students of using it without any real proof in many cases, and it’s a somewhat troubling environment.

Racheal: Yeah, I think that’s such a good point, John, about the range of approaches that students are taking to AI, as well as the different assumptions that faculty members are bringing. And I think that there’s sort of a spectrum of assumptions, where, on the one hand, you have the idea that AI should be completely left out of the classroom and any use of it is cheating. On the other end of the spectrum you have we should embrace AI and use it for everything in the classroom, and if we’re not teaching that, we’re being irresponsible. So we are seeing very polarized views on our campuses, and I think that a professional development opportunity like the one we’ve been running needs to respect and honor and speak to both of those positions, while also making the conversation a little bit more nuanced. And I think one of the ways that we’ve done that is to show that bringing AI into the classroom actually addresses the problem of students using it for cheating and plagiarism, because students are: A) aware that the faculty member knows about and is actively engaged with those tools, so they might be less likely to just copy and paste, which is the use that most people do regard as cheating or plagiarism. And in addition, it not only shows that faculty members are engaged with and knowledgeable about these tools, but in my experience, students are really hungry for guidance from faculty on how to engage with them in an ethical and informed way. So we’re helping them build those AI literacy skills by confronting and talking about these tools in our classes. And to me, that might lead to them using the tools more in specific ways that we’re guiding them to, but it can also lead to them using them less in situations where we want to protect those learning objectives. Like Stephanie, I teach in the writing classroom, and I have been devoting this semester to engaging with and thinking about AI, but that has meant we have plenty of writing and activities where I’m asking them not to use it, and then we have activities where I’m asking them to use it in specific, guided, critical ways. And so I think that that more nuanced approach than we need to totally keep it out or we need to totally embrace it, that’s the area that a lot of our faculty are working in, and I think it serves our students really well.

John: And Stephanie and I have also, in a series of workshops here, encouraged faculty to co-develop their AI policy statements with students, so that students are involved in that conversation, and if they help develop the guidelines for the class, they’re much more likely to abide by them and to buy into them.

Stephanie: To piggyback off of that as well, in our cohort, especially, I mentioned earlier that we had a range of experience levels with AI tools. And what’s been really interesting is that for participants who are trying these tools for the first time, they’re having some important realizations about what these tools are or are not capable of, and I think that’s really important. If we are teaching and we’re saying, here is my AI policy, and here’s why, if we are familiar with what the AI tools can do, I personally feel that our students are going to be more receptive to that. If we are making assumptions about what the AI tools can do, or how well they can complete a task, but we don’t really know, then how are we going to recognize an AI’s work when we come across it? So for example, this is a dated example, but I think it works still really well, in a writing classroom, for a little while after ChatGPT came out, the solution to avoiding AI-generated work was to ask students to write reflectively, because they’ll be writing from their own personal experience, and an AI tool can’t replicate that. And I remember leading a workshop, and I showed an example of a reflective essay that was generated by AI, and this was like two years ago. All I had to do was prompt it a little bit. I asked it in a revision of the reflection to include a sad story about how the tool had a childhood dog that they really loved, and that relationship with their dog is the reason that they wanted to pursue veterinary studies. And it generated this beautiful reflective essay that you would have never known from anything else. As I continued to prompt it, I asked it to include volunteer activity, and it made up a name of an animal hospital… at the time that didn’t exist, but now it would, because AI tools have gotten so much better. So it’s just really interesting as we think of what the limitations are, and how there are not as many limitations in so many different fields. That is, of course, not the case for all fields, but the tools continue to evolve so quickly. So I think it’s important for us to recognize what those limitations and capabilities are, and then if we know, I feel that our students are more likely to trust our judgment, and then believe us when we say, “Yes, perhaps you could use AI for this, but let’s not because I want you to develop this skill that will help you as you continue taking more classes here, or as you graduate and start looking for a job.”

Rebecca: We’ve talked a bit throughout the conversation about how faculty have engaged with the program over the course of the year so far, in creating assignments and in the discussion boards, et cetera. Is there any additional things that you want to share about how faculty have responded to participating in this program.

Racheal: I think that some of the feedback we’ve received is that it’s been really nice to have institutional support to help faculty do work that they felt the need to do in their classes. Faculty want to respond to AI’s emergence, and oftentimes we know faculty are overworked. There’s so much to do, and so being able to get that institutional recognition through a stipend to say “this is important, and we’re offering support for developing these activities.” We’ve heard faculty report that they’re happy that the opportunity is available to do things that they’ve already wanted to be doing, but here they can see the institution supporting that.

Stephanie: I agree, and it’s also been very helpful as we have these conversations with our participants to think about effective practices on the whole with AI. So for example, there has been conversation everywhere about using AI tools to grade student work and to assess student work. So we’ve had opportunities to have those sorts of conversations about why, perhaps, it’s not in our best interest as teachers to use AI to give feedback to student writing, for example. And it’s been nice to hear what other people are doing, like Racheal said, with that support aspect, but also the conversations allow us to sort of come across those topics that might just be neglected otherwise.

Racheal: And I’ll just jump in, Stephanie, because your comments about using AI to assess and grade has also been an important part of our conversations at Oneonta. And again, we’re seeing really polarized views, where some people are saying, “I would never do that.” Some people are saying, “I’m already doing that in these specific ways.” And we’ve talked there, for example, about the Student AI Bill of Rights, which circulates online. And one of the elements of that AI bill of rights for students is that students should have the right to know when faculty are using AI for feedback or other forms of engagement, and so I agree that this has been a really great place to have those conversations which might be taking place informally or in departments, but in this interdisciplinary setting, people are able to talk about those issues and share their views.

John: So we always end by asking, and this is something particularly appropriate with AI, what’s next?

Racheal: As we’re wrapping up this project, we are looking to move the learning activities that our faculty will be submitting to Brightspace into an open access repository that my colleague Ed Beck here at SUNY Oneonta is helping to develop, not only for our grant, but for another IITG-funded project out of Albany that is generating some K-12 resources. So we are creating that repository space via SUNY Create and after we collect our faculty learning activities on Brightspace, we’ll be guiding them to move revised versions of those activities into the repository. So we also have our presentation at CIT, which is the Conference on Instructional Technology that SUNY convenes every year across the system, we will be presenting at CIT on this project, and by that time, we will have more information on the website and how to access it, so that faculty, not only across SUNY, but across the internet, will be able to view the learning activities that we’ve been talking about today.

John: And we should mention that SUNY Create is basically an instance of a system-wide license for Reclaim Hosting’s domain of your own.

Racheal: And after CIT, we’re hoping to extend this work, maybe in new ways. We’re thinking about ways to bring campuses together for communities of practice. And one thing that’s been really important to me about this project has been the value of working with colleagues in educational development, whether we’re talking about faculty or staff at all of our six campuses, all of whom have brought different skills, knowledge, expertise, passion to this project. We’ve had leadership team members who work in the social sciences and have brought that qualitative research and assessment knowledge to the project. We have created pre- and post-assessment surveys that are helping us collect data on the success of the project, and we couldn’t have done that without Deepa Deshpande, who’s at SUNY Alfred. We also have leadership team members who are bringing expertise in the classroom, that includes Stephanie, John, and Laura Pierie, who is with us from Morrisville, and having that faculty insight from a range of disciplines has really grounded the project in the needs of our participants. We’ve also had Dana Salkowski and David Wolf join us from community colleges, and having their input as directors and leaders in teaching and learning has been really crucial to helping us think about how our project could serve faculty at all of our different campuses that we’ve been working with. So I think that it’s been a really amazing group effort, and I’m so grateful for the leadership work of that entire team. I think together, we’ve been able to really support faculty in some of this work, and I’m excited to think about the collaborations that we might be able to pursue going into next year.

Rebecca: Well, thank you so much for joining us. It’s been a great conversation, and it’s a great project to share with others that they might want to implement something similar on their campuses.

John: Thank you. It’s been great working with both of you and I’m looking forward to future collaborations.

Stephanie: It’s been wonderful chatting with you all.

Racheal: Thank you for having us, and it’s been wonderful talking with you today.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]