348. Active Learning Initiative at UGA

While there is compelling evidence that active learning results in increased student learning, these initiatives often face resistance from students and faculty. In this episode, Megan Mittelstadt and Leah Carmichael join us to discuss the active learning initiative at the University of Georgia that provides professional development for faculty, active learning training for students, and for the redesign of classroom spaces. Meg is the Director of the Center for Teaching and Learning at the University of Georgia. Leah is the Director of Active Learning, also at the University of Georgia.

Show Notes

Transcript

xJohn: While there is compelling evidence that active learning results in increased student learning, these initiatives often face resistance from students and faculty. In this episode, we explore a large-scale active learning initiative that provides professional development for faculty, active learning training for students, and for the redesign of classroom spaces to better support this initiative.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Megan Mittelstadt and Leah Carmichael. Meg: is the Director of the Center for Teaching and Learning at the University of Georgia. Leah is the Director of Active Learning, also at the University of Georgia. Welcome Meg: and Leah.

Meg: Hi.

Leah: Good to be here.

Rebecca: Today’s teas are:… Meg, are you drinking tea?

Meg: I am, I’m drinking double chai. Tea is a big deal. In our center. All of the people who work here have favorites. And we provide tea service at every workshop and I checked this morning, the most popular tea at our workshops is lemon ginger.

Rebecca: Oh, that’s good to know. Another teaching center full of tea. [LAUGHTER]

Meg: Yeah, we even have the teas of folks who don’t work here anymore that we keep stashed, just in case. So, Leah, you know, Colleen. Colleen drank…

Leah: Yeah.

Meg: …caffeine free peppermint tea, and we still keep it stashed, even though she doesn’t work here anymore. [LAUGHTER] Just in case.

Rebecca: How about you, Leah?

Leah: Well, I am a diehard coffee drinker. And so I am enjoying a cup of coffee. But my best friend is a potter, so I have her beautiful mug here to enjoy it with.

Rebecca: Wonderful. How about you, John?

John: I’m drinking a black raspberry green tea today.

Rebecca: Oh, that’s nice. And I’m back to my blue sapphire black tea.

John: Black and blue?

Rebecca: It’s black and blue. [LAUGHTER] We’re recording this at the end of the semester. [LAUGHTER]

John: We read about the active learning initiative in a recent Chronicle article and we invited you here to talk a little bit about the active learning initiative at the University of Georgia. We were really impressed to see that the university had budgeted over a million dollars a year for five years as part of the university’s Quality Enhancement Plan. This is a really large investment in effective teaching practices, and it’s especially impressive that this is happening at an R1 university. How did this initiative come about?

Meg: Yeah, we’re really excited that this investment is happening. And it kind of stems from an initial pilot that happened in 2018. There was a presidential initiative on active learning that resulted from a faculty task force. And so in 2018, our president allocated a million dollars to active learning programs, as a pilot. I feel super lucky to have a president who cares so deeply about the teaching enterprise at UGA. It’s something that you don’t see at every institution. And so I really appreciate it, value it, and count my luck there. Our president allocated a million dollars in 2018 for an active learning course redesign institute and a pilot classroom infrastructure renovation, and also the creation of a one-credit course for our students called Becoming an Active Learner that’s deployed by our Student Success Center. And that first year, we thought, “Gosh, it might just be a one-year investment, so we really got to make this count.” And so all the folks in the center sort of got together and backwards designed the programming for our first Active Learning Summer Institute, which by the way, was one of the most gratifying and enjoyable experiences of my career to get five educational developers in a room and get to go totally meta and apply the backwards-design process to the creation of a Course Design Institute. We nerded out. [LAUGHTER] We had like the room where we had all of our notes on the walls, and it was sort of our Beautiful Mind room for the Course Redesign Institute. We loved it. And so we designed this institute for 2018, and it went quite well. The folks at our institution who applied were way more applications than we could accommodate in the Institute and the folks who got into the Institute had a good experience. And we were able to show a couple of courses that had been redesigned seeing some really significant positive outcomes for students, both in terms of their depth of understanding, but also in some of the metrics in courses like numbers of Ds, Fs and withdrawals from courses decreasing in a few really key courses like our introductory mathematics course sequence. And so we were able to go back to the President and the Provost every year after that and say, “Okay, here’s how the course redesign Institute went, could we get funding to do it again?” And so we continued to do it every year. And this continued for a couple more years. And when we got into the 2020 timeframe, UGA was preparing for our next reaffirmation through SACSCOC. And for SACSCOC reaffirmation, an important component of that reaccreditation proposal is the creation and proposal of a Quality Enhancement Plan (or QEP). And so a committee was assembled to come up with what that topic might be, and they looked at programs that existed on campus as well as the university strategic plan. And luckily, by this point, our strategic plan had been drafted with quite a few key performance indicators related to the adoption of active learning practices. And so they identified this opportunity to scale the Active Learning Initiative at UGA. And so they proposed this topic, it was selected and now the funding has been stabilized for some of the pilot programs that existed, but also some newer initiatives to0.

Rebecca: There’s a substantial body of work supporting the efficacy of active learning and increasing student learning and reducing equity gaps. And it’s so exciting to hear about such a big investment and the interest of faculty [LAUGHTER] having more interest than spots for example, But I know one of the things that sometimes happens when faculty decide to start implementing active learning in their classes is sometimes there’s some pushback from students, but I heard you say something about a one-credit course for students. Can you talk a little bit about that? And how that ties to preparing students for this endeavor?

Meg: Yeah, so because we had this course from the get go, there was already this idea that reaching out directly to students and building their capacities and dispositions as learners to amplify their success in an active learning environment is something we might ought to think about. Of course, we also think about it from the faculty side too. One of our most popular sessions in the Course Redesign Institute is a session on student resistance to active learning, and we rely heavily on Tolman and Kremling’s Why Students Resist Learning for that session and then talk about some of the instructor facilitation and explanation strategies that instructors can use to ensure that student resistance is not cropping up and to help prevent barriers to student engagement. I think what’s really cool about Tolman and Kremling’s book is that they reframe “student resistance” to “barriers to engaging” and so we work with faculty to reframe student resistance as “Where are the barriers to engaging that we need to address as instructors?” And then on the student side too, the philosophy is “Where are there barriers to engaging?” And then “When can we do some really important work to help students appreciate that engaging in active learning is a really actually durable practice for them?” It’s really going to benefit them long term. And since that idea was already in the mix, as we were coming up with the programming for the QEP, there was a really large committee, I think, a 33-person committee of faculty, students, and administrators who proposed the programming that would go into our Active Learning Initiative. And one of the things that they wanted to think about was the role that metacognition plays in active learning. That was identified really early on because we were already aware of the student piece, thinking about metacognition as a key component of active learning. The committee spent quite a bit of time unpacking: “What are the learning dispositions involved in student metacognition? And what are the learning dispositions involved in lifelong learning and other benefits of active learning? And where’s the overlap?” So we looked at things like the AAC&U value rubrics on lifelong learning. And I think the other one was, I want to say, it was like transferable learning, I’m getting the title wrong. But anyway, there were two AAC&U value rubrics that we looked at, as well as some of the literature on metacognition and literature on lifelong learning. And our assessment team identified four student learning dispositions that seemed to exist across all of the categories that we were looking at: students acting more curiously, students engaging with initiative, students becoming more reflective in their practice as learners, and students drawing connections between what they were learning in their active learning courses and developing other knowledge and skills. And so we use this framework of these four learning dispositions to really drive our student programming. And I’m going to pass it over to Leah, so she can share a little bit more about how we’ve expanded beyond this one credit hour course to a whole portfolio of student facing initiatives.

Leah: So, first of all, I would like to say with the instructor development piece, I was an instructor on campus, a lecturer, and was in that first cohort in 2018, an international affairs scholar. And it really was the first time that I had had such a professional development opportunity that felt like it truly revolutionized the way I taught, the way I thought about my students, the way I interacted with my students. And so whereas I was taught to transform one of my courses, I ended up transforming them all. And when I was able to be on the team for active learning, I signed up, it was really excited to do that. So it was an amazing experience. For the student piece, what I really love is that we have not just gone to the instructors and said, “Alright, it’s time to consider something different and then your job is to not only do all this hard work to change and rethink your courses, but then can you do us a favor and sell it to the students?” That’s just such a burden and very difficult. And so within CTL, they do a great job of training us in the Institute to go in first day and frame the course, to craft our syllabus language in a way that would potentially be able to introduce this course before even day one and to help decrease that student resistance. So that was a really important piece. But we also would have these teams of students, these just superstars on campus, who are also part of the programming for active learning. One of my absolute favorites is the peer learning assistants. This is an on-campus paid position for students. They apply and they are able to either return to a class or return to an instructor whom they have worked with before. And they attend a class where they facilitate active learning approaches and activities with the instructor. So the peer learner is responsible for helping with the planning of the course and preparation with the instructor implementing it and then also of course, following along with the course materials. And this is a key: this near-peer experience has been key in allowing the burden to be shared in introducing active learning within a classroom. And the students look at these peers, maybe a semester ahead of them sometimes, maybe years, and they start to see these mentors that are built in. And these students know and care about active learning. They see why it works in the class and they’re returning to the class. And we have found that those classrooms have been just the most wonderful place to see this cultural change among students. We also have active learning ambassadors. And they are trained to do outreach. They do demonstrations for faculty. One of my favorite moments is we have an active learning summit every year open to the public. But we have an active learning summit in which they run a showcase where the faculty walk in, and there are students behind tables with interactive material demonstrating different active learning techniques and how they see it in their class, how they use it for study skills, and how they would enjoy instructors being able to incorporate it in their classroom. The faculty love it. They stop, they ask questions, they think, “Well, what about this? Well, this would be difficult.” And then students give that experience from their perspective of how these techniques work for them. And we see that the faculty really crave that and really enjoy being the learner in that moment, and learning from a student about what techniques work best for them.

Meg: And Leah, I think one thing that’s really unique about our quality enhancement plan that I haven’t seen at other institutions yet, or if they have them, I haven’t seen them, so mea culpa. But one thing that’s really cool about our Quality Enhancement Plan is that we have a partnership with our Office of Student Affairs where they’ve designed some programming through our residential life curriculum to embed active learning in the co-curriculum too, so that students are experiencing and talking about the value of active learning not just in the classroom with each individual instructor, but experiencing active learning outside of the classroom in the co-curriculum. Students who are leaders on campus are trained in active learning facilitation techniques, so that when they go in to serve as leaders of student groups on campus, they have some of these active learning tools in their back pocket that they can deploy and experience on the other side as well. So we’re really seeing active learning proliferated, not just in the classroom, but outside of the classroom too.

John: It’s nice to see such a holistic approach. In so many cases where we’ve seen active learning initiatives, it’s usually just designed for faculty development, to go and fix all those issues and challenges. And by approaching it from all these angles, it seems to be a really effective way of addressing some of the challenges that faculty face when doing this. One of the challenges, though, that faculty sometimes face when they introduce active learning is they go into a classroom where the seats are either bolted down, or are very rigid. I think part of your plan involves some type of infrastructure changes to support that. Could you talk a little bit about how classrooms have been redesigned as part of this process?

Leah: It’s such a key aspect of it, I have the anecdotal experience where I graduated from the Active Learning Summer Institute, went into a classroom ready to be highly engaged and interactive and set up have a highly mobile class design only to find that I was in stadium seating, with fixed seats and I thought, “Oh, no, what am I possibly going to do?” And so we have three foci, within the initiative. And we’ve mentioned the instructor development piece. So key, but as you said, it can’t stand alone without support for the other two. We have the student piece, we also have the classroom piece. And so our goal is to every… usually it has to be in the summer, when it’s much quieter… every summer, we just push for classroom enhancements across campus. And we have three criteria in mind that we see as not necessarily perfect for every single classroom situation, but the baseline for which we want to capture for an active learning classroom. And so the first is: does this classroom have mobile furniture? Are we able to move students around? Could an English professor have a roundtable discussion? And then within the 20-minute transition time, could my students go into that same classroom, which happened last semester, in a highly mobile classroom and be able to set it up into a simulation of roleplay thing for my international politics course? So do we have classrooms with that mobile furniture is the first thing. The second is, do we have student collaboration tools? And these do not have to be high tech. In fact, we ran the survey, and students and faculty said just give us more dry erase boards. We don’t need the fancy, fanciest of smart screens. Please just give us the basics. But let us work on a project huddled over together and be implementing and creating something new. And then finally, it’s just general space, do we have enough space that we are able to not only move the furniture around, but move people around in an effective way. And so 25 square feet per student is kind of our ultimate goal. And so we looked around and I was actually just running the numbers recently. And I think on campus we have, depending on how you measure our classrooms, between 400 to 600 spaces, it depends on if you count labs, and these other learning spaces, about 82% have at least two of these criteria. And so we’re getting there. We’ve made a real push and we’re trying to say: “Yes, instructors, please change but also know that we’re going to meet you with an environment that facilitates this kind of approach to learning.

Meg: Well, I also have to interject and say that the stance of the CTL is that active learning can happen in any classroom no matter the features. And so of course, like a good backwards design course or re-design institute, we always start out with the situational factors of a course. And that might include what’s the likelihood that you’re going to be in a fixed-seat auditorium or fixed table, but moveable chairs, and think about the choreography of that, the active learning techniques that you’ll adopt based on that. But in Leah’s example, in particular, if I remember correctly, Leah, you had a student in a wheelchair in your course the first semester after you redesigned your course. And when you were in the tiered auditorium, that meant that there was a specific area where the student in the wheelchair could sit and they couldn’t move through all of the case study roles that you had available. And so you were able to advocate for a different classroom, you obtained one, thank goodness, and everyone could move through a flat classroom instead. And so sometimes things crop up, even if you have planned to be in a fixed-seat auditorium, that you have to be able to modulate on the fly based on the characteristics of your students each semester too.

Rebecca: Can you talk a little bit about whether or not your program has focused on specific programs or if it started with specific programs and its spread to different programs or what your strategy is for involving faculty across disciplines?

Leah: One of the most important parts that I really liked about this initiative is it’s not a mandate, we are not requiring any instructor to adopt active learning through a mandate. Instead, it really is a support, educate, encourage. And so we really have not been focused on okay, this unit by X year needs to have Y outcome in terms of an active learning goal. But in terms of strategy, what’s nice is we can start to see the people that walk in the door to find the active learning Summer Institute. And the high demand that first year was a nice little kind of bellwether that, okay, there is something going on here. When you have more people that can fit, and that’s consistently. The CTL needs to keep it kind of a quality curated experience with a cohort that goes through and has to do these really difficult re-education, understanding education differently, and then of course redesign. Such a dense set of tasks. So with this cohort like experience really requires so much of the faculty. What we started to notice, as we looked around and realized that instructors from different schools and colleges were coming to the Summer Institute, and we have seen that as the CTL has expanded beyond the Summer Institute and provided workshops, both standalone workshops, as well as a course redesign experience outside of the institute. We see hundreds of faculty each semester coming into the programming of active learning. And really we have, I believe, 19 schools and colleges that have undergraduate programs, and we have representatives from each that participate in this programming.

Rebecca: That’s wonderful.

John: What type of training do faculty receive as part of the institute and other training that you provide?

Meg: It was initially just that summer Course Redesign Institute. And what we started hearing from faculty as we would get these applications every year is the deadline would pass, and we’d hear from some faculty who would say things like, “Oh, I really, really want to do the institute. but summer is my field research season” or “I’m in the clinics in summer,” or “I have childcare duties that I can’t get away during that time of year.” And so there started to be some sort of advocacy for other on ramps into the programming, so that when we had the opportunity to stabilize the funding through the Quality Enhancement Plan for active learning initiatives, we started thinking about how can we make this program one that faculty could find the on ramp that is appropriate for their needs? Is it a faculty member who’s been using active learning for a long, long time and they really want to brush up on their skills in one particular area? And so that might look like engaging in a specific workshop on something like facilitating group discussions in a large lecture environment, or looking at student metacognition and how that plays into a particular student developmental level that they might teach in their courses? Or does somebody want to do the whole workshop series and Course Redesign Institute all the way through, but they can’t do it during the summer, or anywhere in between. And so what we ended up with is a workshop series, we have 12 standard workshops that we offer over time: four fundamental workshops, and eight special topics, workshops, and we rotate through those. We offer a couple of each every year. And then we have the active learning summer institute that happens every May. And then starting next academic year, over the course of the spring semester, we’ve designed a course redesign experience that takes the course redesign elements out of the active learning Summer Institute and decouples those elements from the active learning workshop components of the Course Redesign Institute that we currently have, so that faculty can upskill through the workshop series. And once they’ve taken eight of the 12 workshops, they can apply for the course redesign experience over the spring semester. Through that course redesign experience, they come and meet with us for a couple of hours every three weeks or so over the course of the spring semester. And the idea is that by the end of that course redesign experience and workshop series completion, they’ve recapitulated all of the experiences that you might have during the multi-week summer Course Redesign Institute. We also provide different communities for folks to engage with. We have several robust Faculty Learning Communities on campus. So we have multiple faculty learning communities related to active learning, one specifically coming up this year that Lea generously funded from the Active Learning Initiative budget for folks who are looking to engage in SOTL, Scholarship of Teaching and Learning, on their active learning endeavors, as well as specific faculty learning communities for the folks who have peer learning assistance in their courses, or who are engaged in the work after the course redesign experience, for the alumni of the Course Redesign Institute. We also have opportunities for faculty who are highly engaged and master teachers to serve as active learning mentors to their peer faculty. So we have three active learning mentors every year who work with the folks who’ve graduated from the course redesign institutes to give them opportunities for office hours with their faculty peers for micro mentoring sessions on specific topics of interest. They also create supportive resources. We have similar mentors for our PLA faculty, our peer learning assistant faculty. And for those who are serving in those mentoring roles, the Center for Teaching and Learning provides them with opportunities for additional coaching and support on instructional development, educational development, instructional coaching practices that they might bring to bear in those consultative sessions with their peer faculty, too. So the goal here is that we have some workshops that are entry level, some special topics, workshops, the course redesign experiences, some stuff for the master teachers, who are really wanting to be mentors and wanting to kind of serve as role models within their spheres of influence and for others who are moving through this programming, and we’re helping them build capacity in those areas, too. And then, of course, we have faculty on campus who have been working in active learning for years and years and years, and who published on active learning, who are real role models nationally. And so we really try our best to honor their expertise and bring them in to help frame the ways in which we deploy our programming, but also to serve as guest speakers and to engage with folks who are newer to active learning so that we can help provide opportunities for that cross-collaboration and multidisciplinary discussion.

Rebecca: I love the range of options you have available, and really how much you all have embraced the idea of flexibility for your teacher learners, just like we might want in our classrooms as well.

Meg: Yeah, yeah, that’s right.

Rebecca: Can you talk a little bit about some of the active learning techniques that faculty have adopted after going through the institute or other variations?

Meg: Yeah, so as you’ve heard now, our summer institute is three weeks long. And so not only are we giving sessions on active learning, but we’re modeling active learning throughout. And so folks see a lot of different techniques that they might not have seen before. And they’re trying to identify which things that they’re seeing, which instructor moves they see, that align with the needs of their students, of the needs of their course, and their teaching persona as well. So as you might imagine, folks to leave the institute, select different things that are authentic to their practice and their situational factors for their course. So we go into the Institute, which actually begins today for summer 2024. We’re recording on the first day of the Active Learning Summer Institute. So we’re getting the anxious energy of meeting the new faculty. We start out on the first day of helping faculty appreciate that we don’t see active learning as a homogenous set of teaching activities, we’re not going to give them the recipe. We want them to discover the recipe that works for their course. So our QEP is sort of predicated on the notion that the extent to which students are actively engaged in thinking and applying what they’re learning is of far greater importance than the particular active learning instructional strategies used. And so to that end, the course redesign process helps faculty first hone in on the key enduring understandings and learning outcomes at the course level they want their students to leave with and identifying key learning bottlenecks or threshold concepts that they’ve identified that students tend to either have a tough time with, or they’ve really had to work hard to help students grasp that concept in order for them to move from novice to expert in that field. And then we encourage faculty to design their active learning interventions in alignment with those key features of the course so that active learning is more like the trunk of the tree rather than the ornaments on the tree. And our philosophy there is that when pressed for time, the ornaments are easy to cut, but if the act of learning is baked into the backbone or the trunk of the course, that’s just inherent to the identity of the course and it’s not something that’s going to be cut for time. And also you’re putting your energy into where the biggest impact is going to be for your learners..

Leah: If I may add, this was the revolutionary part of being a participant in the Active Learning Summer Institute. I thought I was going for the tips and the tricks, I’d get my little suitcase of ideas and I would just kind of put it all in there through the Institute, and then graduate and use all those as those ornaments as Meg: described it. And instead, I found myself looking at my course and realizing that the backwards design principle had me throwing a lot of content at my students, and not really aligning the activities in the course towards the student learning outcomes. Once I was able to do that hard work, then the excitement of “Ooh, okay, well, which tips and tricks, which kind of active learning engaging styles would be most useful for this.” And that’s just a really exciting way to do it as an instructor.

John: Most instructors, as they come through their educational process, have not been exposed, particularly not in graduate school, to active learning techniques. How do you address issues of faculty resistance to completely changing the way in which they’re approaching the design of their classes.

Leah: It takes a while, [LAUGHTER] having been one of the resistant faculty, I’ll speak to it from this perspective, I think Meg: referenced the former colleague Colleen, and she was my mentor going through this institute. And I remember her asking me to write all the content that I felt was so essential to get the students to to learn throughout the semester, and I distinctly remember this moment where she said, “Okay, let’s put that in an envelope, all that stuff that you just wrote down, we’re just going to put it in an envelope, and we’re just going to set it over here, it’s still very much matters, don’t worry, it’s not going away. It’s all still important.” And then she said, “Let’s really think about the future student. Who are you teaching? What would be a success?” And I think that’s a moment where I could take a deep breath, as a faculty. At first, it was scary and fearful. But it’s a moment for the first time you take a deep breath and go, if I miss a case in international law, they are still going to thrive in this world. I’m focusing on what skills they actually need to thrive in this future career. And so it’s not about that specific case, it’s about the skills. And once I was able to realign that, and it’s so important to be sitting in a cohort, it’s not an isolating experience, you’re sitting in a cohort with other people doing the same reformulating of their whole profession, with mentors from the CTL, helping and encouraging them. Without that sort of ecosystem, it would be a very lonely and harder… just slow… process, maybe, even if you are willing to change. So I think the quality of that instructional development experience within the CTL cannot be replicated. And then I hope the other two kind of prongs of the initiative, once they leave the institute, they’ll still have the mentorship for the next year with CTL and with the faculty mentors. They then can leave and say: “We have the classroom engagement things.” We are also trying to change the environment to help you and we want your feedback. We are always asking those active learning instructors, what can we do differently within these classrooms? And then also the students… we’re engaging the students. So you’re going to walk into class, and they’re going to know what you’re trying to do, and that it’s recognized as a really good thing to do across campus. So it’s the ecosystem that will create more empathy and support and therefore reduce the resistance.

Meg: Yeah, so Leah mentioned, this idea of you have this mentor that exists for the year after the institute. And I don’t know that we explained that. That’s a pretty unique feature of our Course Redesign Institute. And it comes somewhat from our Center for Teaching and Learning has quite a bit of tradition around investment in longitudinal cohort-based programming. So we have a lot of one-year, two-year, multi-month cohorted faculty development experiences, because we find that that interdisciplinary conversation is so helpful for helping folks build connections and shared disciplinary content, and teaching practices with one another in ways that proliferate those great ideas across campus and seed innovation. And as we thought about the ways that our faculty fellows…we call them Faculty Fellows Programs at UGA… ways that our faculty fellows programs were successful, were in that sort of longitudinal support that was available. And we wanted to replicate that as best possible with our Course Redesign Institute. And when we thought about the outcomes of our institute, and how we wanted people to leave the institute, be prepared to deploy a redesigned course in the coming academic year. And we started thinking about what that experiences like as a teacher, and we thought, “Gosh, people are going to leave the institute, be all excited, go back to their department, where they might be one of few who’s engaging in this sort of robust active learning commitment, especially in the early years of the institute.” How can we support them in being successful when they go into their first class and leave the class and go: “Ooh, that didn’t feel like it usually felt I wonder if it went well?” or they do their first robust activity and maybe it doesn’t go quite as they were expecting? How can we be there for them in those moments? So as folks leave our Course Redesign Institute, they’re each paired with a mentor from the CTL. And that mentor is their touch point for the year following the institute, so that when they have that class session that throws them for a loop or they get some feedback from students, that they’re not quite sure how they’re To respond to it, they have a touch point that they can go to. And it’s someone who knows their course intimately, who’s worked with them through the redesign process, is aware of the ways in which their course has changed, and is there to provide them with that support. And also that mentor is the person who performs… we call them mid-semester formative evaluations… but at other CTLs they’re called small group instructional diagnoses. So at the midpoint of their course, someone from the CTL comes in and talks with their students to focus group them, to collect some feedback on how the course is going so far, and then works with the faculty member at closing the loop on that feedback to help students appreciate that they are such an important component of the redesign process too, because the instructor is engaging in this two-way exchange of information with the students both on the content and on the operational organization of the course and refining the course in response to that feedback. So we want the students to appreciate that their experience and the way that they’re learning and what they think could be happening to support their learning even better, how that could be incorporated into the course as it continues.

Rebecca: I really love the depth of the culture change components that are built into this initiative, from feeling like you’re part of something bigger, the fact that it’s voluntary, your support structures, it’s holistic, both Academic Affairs and Student Affairs, there’s opportunities for feedback loops. Change can be really difficult, but when you’re a part of that bigger network, it can be helpful. The timing of this was really interesting when it kicked off…so, a couple years before COVID, so you had some time to establish the roots, and then COVID erupts, AI erupts, there’s more first-gen students, we’ve got a lot going on, and there’s a lot of change happening. How do you help your faculty avoid burnout with so much change and maintain this momentum?

Meg: Yeah, great question. You’re taking me right back to the pandemic and that experience of everyone just doing the best emergency remote teaching you could do. And one of the things that I remember from that period is faculty who had gone through the Active Learning Summer Institute mentioning that they were so glad that they’d gone through the institute because it made pandemic teaching easier. They had recently conceptualized of their course learning outcomes that identified the most important content that they wanted to prioritize, they had a sense of some lesson plans for how to deploy these things. And even though the institute happens in person, because of the way we were talking about how you would make adjustment to different things in different settings, they could more easily conceptualize of how to engage online or asynchronously with students using active learning principles. So we had this early indication that the folks who’d experienced the course redesign process felt better prepared for emergency remote teaching and pivoting quickly. We also found active learning to be perfectly timed as an initiative as a response to post-pandemic disengagement. So when students were coming back into the classroom and members of our community were craving that deep intellectual engagement and community with one another, but then in some instances, were a little bit disappointed when it didn’t materialize in the ways that it might have before the pandemic, we found that faculty were coming to the institute, saying, “This is something that I’m hoping can meet this need that I have to engage students in rich conversation or to engage students in deep intellectual engagement in this learning community, and how can I address this through active learning?” And also, we’re lucky to have a large number of faculty champions for active learning on campus, folks who’ve been using active learning for a while, and in various departments across the institution, not just in specific pockets. And so they’re serving as role models and mentors to the faculty in their unit. And a message that those faculty champions were helping us reinforce is that teaching with active learning is a joyful way to teach. And so oftentimes, when these faculty champions, these mentors, were talking to folks who were saying things like “post-pandemic teaching is a little bit disappointing,” or “I’m feeling a little bit burned out because of everything that we’ve experienced through the pandemic, and through these various disruptions to higher education.” When their mentors were telling them, “Have you considered bringing some joy into your classroom and making teaching a more joyful experience for you and for your students? And guess what? You can do that through active learning. Let me tell you how I brought joy into my classroom through active learning.” We would see folks come to us and say, “I’m just really looking for some rejuvenation and some revitalization, and I’ve seen my colleagues just transition their course through active learning in a way that they’re just skipping down the hall after they leave their class, and I want that feeling. So I’m hoping that this will be that experience for me.” And so we had people coming to us asking for that, and we hoped that would be the case. And then I’ll just add that like many higher education institutions right now we’re in the midst of an evaluation of teaching culture change, too. And so we’re trying to adopt a more what we’re calling a three-voice culture of teaching evaluations that’s relying upon your individual reflection on your teaching some data from your students and the interpretation and contextualization of that data, as well as peer observation of teaching data and bringing that all together as sort of like triangulated data to provide some insights on our teaching, and then respond to that and have like a assessment loop on this. And so it’s this idea of a culture of continuous improvement with teaching where we’re able to proliferate this message of: “You’re not an award winning teacher necessarily on your first day of teaching, but this is a set of skills that you can develop over time. And also, you’re never done developing it.” And so this culture is helping to amplify the message that taking part in educational development is not just an expected part of being a teacher, but it’s an opportunity to continue your culture of continued mastery.

John: So one of the ways in which this is spread has been from faculty to faculty. One of the things I’ve been noticing a lot on my campus is that it also sometimes spreads from students to faculty. As more and more faculty move from faculty-centered teaching to student-centered teaching, students become a little bit more vocal sometimes about expressing what works for them and that will often result in faculty considering using new techniques that their students have recommended to them. Has that been happening as well at your institution?

Leah: That’s why we wanted to capitalize on this proactively and have channels of growth. And so again, we have little areas in which we have students really playing that intentional role. So rather than maybe a complaining, “I went into this class and I didn’t like it” kind of a customer satisfaction problem area, we really want to make sure it’s more that students, first of all, are trained in the active learning pedagogy. So my active learning ambassadors, as they are chosen, we then say, “Okay, here’s the wizard behind the curtain friends, come join us [LAUGHTER] and see what’s going on behind these intentional classrooms that you really enjoy and don’t quite know why you like it so much.” So we bring them into the understanding of what instructors are doing when they’re in an active learning classroom and why. And then we give them opportunities to then go and front face and tell other faculty, “this is what an active learning classroom feels like for a student.” This is why one of my favorite moments… we have, again, as I mentioned, an active learning summit every year in February, and there was an entire breakout session run by students who were peer learning assistants and active learning ambassadors. And they were talking about the study skills that they have seen through their training, or through their classes that really work in their discipline that they use outside of the classroom to kind of empower themselves to study and to learn. And what was wonderful is that the front of the room were students in a panel and most of the room or faculty and I found that the question and answer period, the instructors would raise their hand and say, “Okay, well, but I’m not in that discipline, but I’m thinking about bringing this into my class in X way, is this something that you would find effective?” And they were asking the students about designing their course to be student centered. And that’s when I think if we put those students that know what is going on, so it’s not a complaint section, it is more a reframing and a sharing out of we know what you’re trying to do and we really appreciate it, and here’s how it works really well, for us. It just starts to kind of change that conversation from being yet another instructor versus student dynamic, which I think we have plenty of those around campuses at any given time to becoming an instructor and student experience.

Rebecca: Having student ambassadors and student voices celebrating this work, is certainly a testament to the success of this work. And you’ve also mentioned some other moments of assessment. Can you talk a little bit about how you’re assessing this initiative?

Leah: There is a rich assessment team who is actually part of the larger Active Learning Initiative. And so I hope I do justice to their very nuanced framework that they have. We have macro level, mezzo level, and micro level indicators that we are trying to capture. At the most macro level demand is one of the earliest indicators of success for us. Are people applying to the Summer Institute? Are ambassadors applying for the program? How many peer learning assistants apply and how many instructors want them, just really kind of those basic quantifiable amounts for demand. And then we also are attempting to capture survey data across the university. We stop the students and say: “We don’t know if you’ve interacted with active learning in the classroom or not, we’re not selecting a given class, we’re just wondering what you would say if you observe a cultural change going on.” So we’re also trying to get these larger macro level indicators. Then at the mezzo level, we’re getting into those lifelong learning dispositions that Meg: referenced earlier, the curiosity, the initiative, the reflection and connection. We have artifacts that we pull from courses that have an instructor who’s either going through all the Active Learning Summer Institute, or who has gone through it already, so we do a pre- and post-analysis. And we have faculty scorers, who can go through these artifacts to see if we are capturing if the artifacts include a prompt about curiosity or a prompt about initiative. And then we are attempting to measure, and we’re working on different ways to do that, but measure if we’re seeing growth over time in these lifelong learning dispositions, and then finally, at kind of the micro level, we’re making sure that the disciplinary knowledge is being enhanced within this that are focused on active learning techniques. One thing we have yet to mention is yet another part of our holistic umbrella. This is why the programming is so large, but we have these active learning change grants. And this felt really good to hear for me as an instructor who went through the institute. And then it was about five years later that the next person in my department went through the institute. What these change grants do is it really allows those who go to the institute to go back to their department or their unit and say, “What if we change several classes at a time through some sort of infusion of deep learning and the CTO will support us, the assessment team will help us assess whether we’re doing what we’re saying we’re doing.” And so on this micro level, the assessment team is really starting to capture some pretty cool indicators, that when you kind of create a unit change, there’s a cascading norm it’s called in international affairs, where you change enough of the unit and enough of the actors in a given scenario, you can then really see that norm start to shift in a unit. And so at each of these levels we’re finding it’s still early. So we’re not saying “It’s a win, close up shop,” [LAUGHTER] but we are finding really robust indicators of early success.

John: Well, you’ve given us and our listeners a lot to think about, and it sounds like a model program that could be copied very productively at other institutions. But we always end by asking: “What’s next?”

Meg: We’ve started to hear from folks that are interested in proliferating initiatives. So, of course, we welcome conversations with folks who are interested in hearing about some of the elements of the program and brainstorming ways to deploy these on other campuses, lessons learned along the way, etc. I mentioned earlier that as folks crave these multiple on ramps, we’re deploying these additional on ramps over time. So starting in the fall, we’ve designed some of our foundational active learning workshops, we’ve taken those and created asynchronous online versions of those workshops that are going to be hosted in our learning management system. So they’ll be available on demand. So if a faculty member doesn’t happen to be available at the time that we’ve offered that workshop this semester, they can still engage, especially in those foundational workshops. And then the spring semester course redesign experience version of our Course Redesign Institute will be deploying for the first time in spring 2025. We’re really excited about that as well.

Leah: I would also say for your listeners, if there is an interest in kind of seeing what we’re up to and helping participate in engaging with this discussion, traditionally the active learning summit has been really focused on UGA community for the first two years, but in the third year, so February 2025, we are going to open up the summit to visitors and encourage them to be a part of these breakout sessions or propose different ideas that they would like to talk about. I know Rebecca mentioned earlier, the challenges of AI, we’re going to try to have a breakout session, though, that focuses on this: active learning best practices in the age of such change. We’ll send out a call for applications in the fall, and we’d love to have anybody who’s interested in learning more, I think it will be Thursday, February 27, and Friday, February 28, on UGA campus, and we welcome all those who are interested in attending.

Rebecca: That’s great. Thank you so much for joining us. There’s so much to think about and so much exciting work that you’ve all done.

Leah: Thank you.

Meg: Thank you so much for having us.

John: Thank you and we look forward to more conversations in the future.

Meg: Absolutely. Anytime.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

 

346. Classroom Assessment Techniques

Classroom assessment techniques can be used to shape instruction to the needs of our students. In this episode, Thomas A. Angelo joins us to discuss the origin of these techniques and evidence concerning their efficacy.

Tom is Clinical Professor Emeritus and Director Emeritus at the UNC Eshelman School of Pharmacy at the University of North Carolina at Chapel Hill. In addition to teaching for 40 years, he has been long involved in professional development and has served as faculty member and Director of teaching, learning and assessment centers at UNC, LaTrobe University in Melbourne, Australia, Victoria University of Wellington, New Zealand, the University of Akron, and Boston College. Tom is best known for his work with K. Patricia Cross on Classroom Assessment Techniques: A Handbook for College Teachers, which was first published in 1988, with a second edition in 1993.

Show Notes

  • Cross, K. P., & Angelo, T. A. (1988). Classroom Assessment Techniques. A Handbook for Faculty. Jossey Bass Wiley.
  • Angelo, T. A., & Cross, K. P. (1993). 2nd. Ed. Classroom assessment techniques. Jossey Bass Wiley.
  • Angelo, T.A. and Zakrajsek, T. (2024). 3rd. Ed. Classroom Assessment Techniques: Formative Feedback Tools for College and University Teachers. Jossey Bass Wiley.
  • Schwartz, Charles (1991). “An Academic Adventure.” March 4. – A description of the origin of the minute paper by it’s originator.
  • Chizmar, J. F., & Ostrosky, A. L. (1998). The One-Minute Paper: Some Empirical Findings. Journal of Economic Education, 29(1), 3–10. (the article that John referenced as his first reference to this topic)
  • Mazur, E. (1997). Peer instruction: A user’s manual. Pearson
  • Mazur, E. (2014). “Confessions of a Converted Lecturer” – a recording of a presentation by Eric Mazur at SUNY Oswego, 5/19/14
  • Fink, L. D. (2013). Creating significant learning experiences: An integrated approach to designing college courses. John Wiley & Sons.
  • Pam Hook and Josie Roberts (2018). “The who, what, when, where, and why of SOLO taxonomy.”
  • Angelo, T.A. (1993). “A Teacher’s Dozen—Fourteen General Research-Based Principles for Improving Higher Learning.”
  • Pre-order for 3rd edition of Classroom Assessment Techniques: Formative Feedback Tools for College and University Teachers.

Transcript

John: Classroom assessment techniques can be used to shape instruction to the needs of our students. In this episode, we discuss the origin of these techniques and evidence concerning their efficacy.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Thomas A. Angelo. Tom is Clinical Professor Emeritus and Director Emeritus at the UNC Eshelman School of Pharmacy at the University of North Carolina at Chapel Hill. In addition to teaching for 40 years, he has been long involved in professional development and has served as faculty member and Director of teaching, learning and assessment centers at UNC, LaTrobe University in Melbourne, Australia, Victoria University of Wellington, New Zealand, the University of Akron, and Boston College. Tom is best known for his work with K. Patricia Cross on Classroom Assessment Techniques: A Handbook for College Teachers, which was first published in 1988, with a second edition in 1993. Even if you’ve never heard of these books, you may have included some Classroom Assessment Techniques in your teaching toolkit. Welcome, Tom.

Tom: Good to be here.

John: We’re very happy to talk to you. Today’s teas are:… Tom, are you drinking tea by any chance?

Tom: No, I was drinking black coffee.

Rebecca: It’s one of our most popular forms of tea [LAUGHTER] as regular listeners know. Today have some English tea time, John. How about yourself?

John: …and I have an Earl Grey tea today.

Tom: Excellent.

Rebecca: We’ve invited you here today, Tom, to discuss the third edition of Classroom Assessment Techniques, to be published shortly. These techniques described in earlier editions of this book have been widely adopted by faculty in all disciplines at all different types of institutions. And a lot has changed in higher education since 1993. But this book is still the go to reference on classroom assessment techniques, which is pretty impressive [LAUGHTER] for a book that was published 30 years ago. Can you talk a little bit about the origin of this book?

Tom: Well, I think the origin of the book goes back to the fact that when I was a young teacher, I recognized pretty early that many of the things that I was doing didn’t work and didn’t work for my students, and that I had to accept that things that had worked very well for me as a student weren’t working for many of them. So I began to experiment with asking questions in a variety of different ways and getting students to jot down answers anonymously. And through that, I really learned more about teaching and learning, I think, than from anything else that I did. And one of the things that I learned was not to assume. So, classroom assessment techniques are things that I’m sure almost every teacher has done. But what we did in the 1980s. K. Patricia Cross and I, was to really formalize those. And we did that in order to create a resource for teachers so that they could find simple tools that would help them ask students the kinds of questions that would help students learn better, and teachers teach better, as they went along. So it really began at that time, and it was kind of an accidental start, in many ways. We were doing it as part of a big project, and what was then the assessment movement, and it just kind of took off.

John: Most of our listeners are probably quite familiar with classroom assessment techniques and have seen them at lots of workshops and have probably, as Rebecca noted before, used them in the past. Could you define what a classroom assessment technique is, for those listeners who are new to the concept?

Tom: Well, in a formal way, a classroom assessment technique is a formative assessment technique. It’s a way of gathering data to use to improve and inform practice, rather than to make judgments. So we have too, most of us, to give students tests and exams and assign papers, and grade their work, evaluate their work, for grades and marks. Formative assessment techniques, on the other hand, are tools that we use to gather data along the way, before those moments in which we grade them, or mark them, so that we can help students get ready and improve and succeed when they’re going to be graded and marked and so that we can, at the same time, find out where they are and how they’re doing in order to help them improve along the way. So they’re really ways of diagnosing, in a way, students’ learning.

Rebecca: One of the most popular classroom assessment techniques described in your book is the one minute paper. There have been many studies of the effectiveness of this technique and it’s been widely adopted in many disciplines. Why was this technique, in particular, been used so extensively?

Tom: Because it’s simple and easy to use, probably, and it takes almost no time. But seriously, the one minute paper, the minute paper was invented, to the extent these things are invented by Charles Schwartz, in the 1970s, at UC Berkeley, and he used it as both a way to take attendance (in his case, he was asking students to jot their names down), but more importantly, to find out whether students were learning from his physics lectures, in his huge physics classes, what he hoped they were learning. And that idea spread through UC Berkeley really fast. And so soon, there were dozens and hundreds of people using these two questions. And the two questions, famously, were: what are the two or three most important things you learned from today’s lecture, in his case, and what questions do you still have the end of the lecture? So students would jot these down on those strips of paper, remember, there was no internet and no computers, really, in classrooms in the 1970s. And then he would collect them, and read through a sample of the 1000 or so that he had every lecture. And from that sample, he would gather ideas about what students understood and didn’t from the lecture, and also what questions were most common. And then he would begin his next class meeting, by going over some of those points that students had had trouble with, or not understood, or missed them completely. Sometimes it happened that a 1000 students didn’t get one point that he thought was critical, and also answering one or two of the most important questions. And what he found was, he could do all of that in five minutes at the end of class, and about five minutes at the beginning of class. And I think it’s that efficiency that has made it so popular.

John: My first exposure to this was a study that was published in the Journal of Economic Education way back around the time of your last edition. I did try to find it, but I couldn’t find the citation for it, but what struck me was that the instructor had used it in one section of the course, had not used it in another section, and found that there was a very substantial difference in learning outcomes between the two groups, which he attributed to the one minute paper. And it’s been incredibly commonly used. But one of the things that really struck me is that at the time when you came out with this book, including the one-minute paper, and so many other classroom assessment techniques, most instruction at the time was the chalk and talk type where people were at a blackboard, maybe a whiteboard, if they were a little bit more advanced in the technology, where most assessment was done with high-stakes assessments. So I think your book was a fairly important factor in helping people shift to more formative assessments. Why have these techniques become so widely adopted?

Tom: Well, I think that one of the reasons why people have found classroom assessment techniques and other formative assessment techniques, there are other kinds, so useful and so important, I think there are several reasons. One is that the student body in the United States and across the world has become ever more diverse in the last 30 or 40 years. And it turns out that no matter who we are, as teachers, most of our students are going to be pretty different in terms of their experiences and their interests and their skills than we are. And that level of difference has probably grown over the years, at least our awareness of it has grown. So I think most teachers now realize that it’s very important to check in on students and see how they’re doing, before we get to the midterm or before [LAUGHTER] we get to the end of the course, and to check in regularly and not to assume that students are understanding our brilliantly clear presentations or the material that we give them, or the readings, or the problem sets, and to really check early on to see if it’s making sense to them, and what kind of sense it’s making to them. So I think that there’s much more realization of the diversity of human learning, and of our students, than there was 40 years ago or 30 years ago. And I also think that as teaching has evolved, especially as it’s become online, people have recognized that we need data. We can’t just depend on students volunteering that data, they often don’t know that they don’t understand. And that that data, used well, can help us and can help them.

Rebecca: I think one of the things that you’re talking about in terms of assumptions is reminding me of some of the conversations that we had early on in the pandemic, where instructors we’re talking about not being able to see the sea of faces to know whether or not students are learning because maybe students aren’t nodding along or providing some visual feedback, but classroom assessment techniques are really important because they allow opportunities to gather that data, which is probably far more informative than the head nodding.

Tom: I think there are maybe a couple of other things that are important about this that people discover. One is that by stopping to ask students to reflect on their learning and respond, you give them time to think. And you prompt them to think about their learning. And from my point of view, forever, the most important thing about doing this has really been to give students tools to become more aware, we could say more metacognitively aware, of their own learning, and to better take responsibility and control or at least direction over their own learning. So what I found is I’ve had many, many, many very bright students who had never been taught any kinds of techniques to think, or to ask themselves questions, or to monitor their own attention, and having those sorts of tools, those sort of metacognitive tools, I think empowers students to make decisions about their own learning, and also to improve their own learning in those areas where it matters most to them, which may not be my course.

John: One of the things you mentioned was that the one minute paper was originally developed in a class with 1000 or so students. And one of the nice things about many of these techniques is that they can scale that way. Now, there was a time when the book came out when this was all done on paper. But now we have so many electronic tools that we can use to do this. Instructors could put up a QR code with a link to a Google form or something where students can give their feedback within a minute or so or we could use polling techniques, and so forth. And many polling systems have built in one-minute questions or some other classroom assessment technique as a default strategy. So this impact has been pretty substantial. And it’s really nice for instructors, not just for students, having that time for reflection, but also for instructors to get some feedback on what students have learned. Because if you just call on students, you may happen to get the only student who understands the material responding to a question or you might get the only student who hasn’t understood the material. But with these formative assessments, you can get a good feel for where students are. And they also get some feel for where they are in a way that they might not if they would just passively sitting there listening to a lecture. So these techniques have, I think, become increasingly more common from the time of introduction. So these techniques have held up pretty well. But you now have a new edition of this book, a third edition of this book. Could you describe some of the changes in the new addition?

Tom: Well, obviously, and you’ve touched on this, technology has changed. And that’s had an impact on teaching, and an impact on learning. So one of the things that I did, in going back to do this book after so many years, was to take a look at first what was out there, what had people published on this. And there are literally about 3000 published articles of various kinds and various quality, about classroom assessment techniques. There were about 300 that I thought were of sufficient kind of quality and rigor to be able to use. And in the chapter on the minute paper, I refer to 150 articles and studied them very carefully. There are many, many books that refer to the classroom assessment techniques. And I also read [LAUGHTER] all of those references. So I took a few years to work through this material and new things were coming in all the time. So one thing that differs is that we have an evidence base across many, many disciplines for what seems to work and what seems to work well. And also much useful information on how to use these with various technologies. So how best to use formative assessment with online, fully online technologies on totally asynchronous classes. And so that’s been really valuable. And those are experiences that I couldn’t have all of those experiences. So it’s terrific to draw in those from other teachers. The other thing is people have invented new techniques or developed new techniques over time. And so in the third edition 25%, or about 15 of the 55 techniques, are ones that didn’t exist in the second edition. And that’s about one quarter of all the techniques in the new book. So one of the reasons why we have different techniques in the 3rd edition is that some of the ones in the second edition weren’t much written about, weren’t much commented on, and they seem not to be much used as far as I could tell. So I’ve picked up the ones that were most used and most written about, and most referenced so that people in different disciplines can find those techniques. And I’ve also found another 15 techniques to kind of fill those gaps and extend it a bit.

Rebecca: Your book offers many, many assessment techniques. I’m curious what your favorite few are that you’d like to introduce instructors to?

Tom: Well, just like I can’t say which of my daughters are my favorite, I do have favorites. I’ll be honest. I have always used the minute paper ever since I learned about it all those years ago, and I learned about it really when I was at Berkeley, in the mid 1980s. And I learned about it, as everybody else there did, kind of by chance, and thought, “Wow, why didn’t I do this? Why didn’t I think of this?” So the minute paper, the muddiest point, which is a variant of the minute paper, which is developed by a really famous and eminent statistician at Harvard, named Frederick Mosteller. And Fred Mosteller heard Pat Cross talk about this, when we were both at Harvard, and being the kind of person that he was he immediately experimented with it. What he found was that it was more useful for him, in his courses, to just ask one question, and that question for him was “What was the muddiest point In today’s class? What was least clear to you?” Many other people have used that as well and written about that. That’s another favorite. A third one is one that I’ve used a lot, being a statistics teacher. And that’s predict, observe, explain, which is often called POE. And that’s one that I’ve found useful in many situations. And I’ve had colleagues who’ve used that even in literature courses, and other courses where they’ll say to students, “let’s stop at this point in the book or the film, and predict what’s going to happen next, write that down, then we’ll observe it, and then we’ll try to explain why that either met our predictions or didn’t meet our prediction.” And while it’s obvious how people do that in science, and it’s part of the scientific method, it can be used in all kinds of circumstances. And it’s one that I’ve used with teachers who I’ve worked with, about classroom assessment and about formative assessment. I’ve asked them to predict what they thought would happen, then observe it, and explain what did happen and whether or not that met their expectations. And that’s been a very useful tool for me, I think, and the teachers I’ve worked with. So those are three good ones. I’ll give you one more, if that’s helpful. And that’s something that we call the directed paraphrase. So three of those four were in the first and second versions of the book, predict, observe, explain is in the third edition, but directed paraphrase simply asks the questions that are sometimes called journalistic questions, who, what, when, where, why, who did what to whom, when, where, why, and how? And it asks students to answer those questions about some phenomenon that they’re observing, or to answer them about an essay that they want to write, answer them about any kind of question. So giving students a framework of questions that they can work through, to make sure that they really understood either a book that they’ve been reading or an experiment that they’ve been doing, has been really valuable to my students. And I’ve said to them, “Look, this is something that you can adapt and take with you anywhere to use to make sure that you understand what’s going on fully. And to find out which part of it maybe you really can’t answer, and need to do some more work on.

John: In terms of the predict, observe ,and explain thing. I took a MOOC that was put together by Dan Ariely on behavioral economics. And one of the things he talked about was how he would often present the results of an experiment where the results were not what most people would have expected. But then he’d asked people for their reactions only after he summarized the results of the experiment. And they all nodded and said, “Well, that’s exactly what they would have thought would have happened.” So he started revising his presentations, where he asked people to make a prediction, and then revealed the result. And it got people much more engaged. And the same type of thing is done by Eric Mazur with his polling technique, where he asked people to answer a question where the question is not obvious and where there’s going to be some disputes. And then once they’ve committed to an answer, they work with other people and he goes through it again, with a little bit of peer instruction. But the level of engagement when students have made some type of a prediction of what they think the answer is, and then they have an explanation of what the answer is either given by the instructor or by other students, the level of engagement and interest just seems to go up fairly dramatically. So I can see how that would be particularly effective.

Tom: Yes, and I’ve learned a lot from the work of Eric Mazur, as have many others. And I think that that’s a critical thing. Because one of the things that we need to help students with, and help ourselves with, is managing our inbuilt biases and that issue of when you hear something new, feeling that you’ve always known it, even though you haven’t, is a cognitive bias that’s common, I think, to all human beings, myself included. So doing something like that: predict, observe, explain routine gives us a chance to really check and see if we know the answer. And then to find the answer, or discover an answer. And then to realize, because we have a document, we have a record, we have data about what we knew in the first instance, to recognize that we’ve learned something and what we’ve learned and that we might have been misinformed or unclear in the beginning. And I’ve suggested to my colleagues, one of the reasons that students don’t value their education as much as sometimes we think they should, or value our course as much as they think they should, is because when they get to the end, they may think “I always knew that, that was obvious, I knew those things.” And so we can help students realize that there’s a start point. And then there are many points along the road to learning, and that they may have learned more than they thought they did in a given course or given program. And I think that’s important for us in terms of demonstrating the value of education. And I think it’s important for students in terms of understanding and developing some humility about how it is that we learn and how hard it is and how long it takes to learn anything that’s really valuable.

Rebecca: I think, in my experience, students really appreciate the opportunity to slow down and reflect on their learning, because they’re often not given that opportunity. And when you’re doing that throughout an entire semester, they really do get excited about how much they’re learning and they get energized by that and seeing that they can achieve things over and over and over again.

Tom: I agree, and what we found with most of the studies of classroom assessment techniques, the simple ones is: first, classroom assessment techniques can’t really be separated from everything else that’s going on in teaching, they’re just one little part of the whole teaching and learning spectrum. They, at best, seem to make about a 5- 8% difference in how much students learn, which is about half a grade. And I think that’s important, but I don’t want to overstate it. So they’re not a panacea and they’re not a silver bullet, but they’re something that can lift up, to some degree, students’ performance and their learning. And the last thing I’ll say is nothing works for everybody. And so there is no technique or no series of techniques that every single teacher or every single student that I’ve worked with found valuable. And so one of the points of having 55 techniques is to hope that there’s one or two or five in there that people will actually find that work for them.

Rebecca: So you just reminded us about there’s 55 techniques, how do you imagine an instructor using your book? Is it a read from the beginning to the end? Do you have it organized in ways that instructors can easily dip in and out?

Tom: I can’t imagine reading it from the beginning to the end, and I don’t think anyone ever has. That would surprise me, except maybe the proofreaders [LAUGHTER] and they weren’t worrying about what it meant. So it’s organized so that there are many ways, there are actually eight ways to find a technique that might be useful for a given situation or a given teacher. One of those ways is the simplest and that’s alphabetical. So if someone has mentioned a technique to you, you can easily find it that way. The others are a bit more complicated, but I think the most valuable for many people is an index of all the techniques that have been used by discipline and documented by discipline, and the examples by discipline in the book. And there are a couple of 100 examples in the book of how people use these in different disciplines, kind of from A to Z, from anthropology to zoology, and everything in between. So those are ways that many people use. There are also a couple of inventories in the book. One is something called the teaching goals inventory, which was in the first and second edition, and has been used in lots of ways by many people and written about and research has been done on the teaching goals inventory, and it is exactly what it sounds like, an inventory of what we think we’re trying to teach. For this book, I created what I think is kind of a more contemporary take on that, and that’s something that I call the course learning outcomes inventory, or CLOI, and it’s really flipping that and saying, okay, it’s important what we want to teach, but let’s look at it from the point of view of what do we want students to learn? And so the CLOI, the course learning outcomes inventory, the statements are statements of learning outcomes. There are 57, they’re divided up into kinds of learning outcomes, and people can use it as a self assessment. Those items both in the teaching goals inventory and the CLOI are all linked to classroom assessment techniques in an index. So if you pick number 47 goal or number 47 outcome, you can find techniques that are linked to that. Lastly, we’ve done indexing by Bloom’s Taxonomy, and by Fink’s taxonomy of significant learning outcomes, and also by the big solo taxonomy, which is one that’s much used in the Commonwealth world, in the British and Canadian and Australian and New Zealand systems. So there are eight ways to skin a CAT in this book, or to find a cat at least. And I hope those will be useful to folks. And I think different kinds of indexing work for different kinds of people.

John: You mentioned that when this book first came out these things were done on paper. But one of the things that’s grown quite a bit over the years, as you’ve mentioned, is online instruction. Could you tell us a little bit about how some of the techniques can be used in an asynchronous environment?

Tom: Well, I think there are huge advantages to the technology that we have now compared to having to do all this on paper, having to read all those papers after class, or all those index cards, in my case, after class and then waited until the next class, when I saw students again, the time lag was a real problem. So being able to respond almost immediately, or in relatively short time to students in asynchronous courses, to their questions, and to their confusions, and to also let them know that they’re learning some things really well… they really got that… and that they’ve come up with some ideas that I never would have thought of that are excellent. To do those things in real time, I think, is much more effective in terms of teaching and learning than waiting two or three days or a week. Also, as you mentioned, we have technologies for polling students and for analyzing their responses that are ever more sophisticated and powerful. And so those save teachers a lot of time and work. I was always going through things counting and deciding on the number of times people had mentioned things on paper. This can all be done and is all done for you if you want to do it. And that, again, allows teachers to focus on what really matters. And that is: what’s the meaning of this data? What are students telling me? …and to make decisions about when they’re going to respond and what they’re going to respond to, because you can’t respond to everything in data if you have more than two students. And most of us did, and do. So I think that asynchronous learning has huge advantages, it has some downsides, but it has huge advantages. And one of those probably is that it makes it more difficult to assume that by looking at students, you can tell that they’re learning or not, which I’ve always found an odd idea, but many of my colleagues at least used to seem to think that if students were nodding and looking at them, that they were just tracking with them and understood everything that they were saying. And I used to say to my colleagues, “I’ve known many smilers and nodders that probably were thinking of something else entirely, and classroom assessment proved that to me.

John: I’ve seen that so much. We’ve heard so many faculty during workshops over the years who say, “Well, I can tell whether they’re learning just by looking at the expressions on their faces.” And I’ve never found that. There are a lot of students who will be nodding and smiling. And then when you ask them a question, or they ask you a question, it’s really clear that they don’t have the basic idea. At an economics conference, I was presenting a paper and there was someone sitting in the first row taking furious notes while I was talking. So I went up to the person after and asked them what they were taking notes about, because I was curious. And they were actually preparing for a session later in the day. But they looked really engaged throughout the whole thing.

Tom: They were engaged, just not with your lecture. [LAUGHTER]

John: Yes.

Tom: So I think the converse is also true. And that is their students, and people, who don’t appear to be paying any attention at all, or who are frowning or who have had their head down, who are actually paying total attention and learning a lot. And that’s one of the things I learned as well, early in my career, was not to make assumptions based on those sort of superficial impressions. And it’s hard not to, but when you actually learn that some students who appear not to be engaged are engaged in their quiet way. And that’s a humbling lesson, or it was for me.

Rebecca: One of the things that I really like about your book, and also the way the collection of the CATS all in one place is the flexibility, the flexibility of navigate the book, the flexibility of options for the instructor, but also the flexibility of options for learners as well, in different contexts. So I just wanted to highlight that flexibility piece that we haven’t really talked about quite yet.

Tom: Well, that’s the hope. And if this is created as an ebook, and I expect it will be, then I think that will offer people even more flexibility and more sort of usability. It will be a more user friendly book. And again, there are huge advantages to the kinds of technologies that we have available now that we didn’t then. So I think the real key to this is one of the axioms in the book. And that is it’s really about adapting, not adopting. So there are many, many tools in there that people have adapted in a variety of ways, renamed, and that’s all fine. And that’s all exactly what should happen. You should find them, adapt them to their uses, and if they don’t like the names of the CATs, they can change the name.

John: Would this be a good subject for a faculty reading group where people from different disciplines could get together and talk about how either they’ve applied these things or how they might apply them in their classes?

Tom: Well, it has been from the beginning. And we began this whole process working with groups of teachers, initially in community colleges and state colleges, who were interested in assessing student learning. And, of course, the point being trying to improve student learning. And what we did basically was use the teaching goals inventory at that time, you could use the course learning outcomes inventory, with those teachers to get them to focus on what they really wanted to know. And in the book, I’ve written about, in one of the chapters, four different ways you can approach using classroom assessment techniques and figuring out which one to use. But everybody had ideas and theories and questions or problems that they wanted to try to solve. And so we started with that. And then we said, if that’s your question or problem or theory or outcome or goal, what might be a good tool to try to help you gather a reasonable amount, a manageable amount, of data from students that might help you learn, and might help them learn? And that’s really where we began, and those discussions and the collegiality that came from them, and the interactions that came from that, were always the most valuable part of that professional development. That when we ask people: “What did you take away from this?” The first thing that they said was not a bunch of techniques. The first thing that they said was the interactions with my colleagues, the relationships with my colleagues, learning from my colleagues, and then they talked about the actual content that they had learned and the things that they had done. So I think those interactions and those relationships that people can make, by having a group focused around this. This is an excuse to get together and talk about teaching and learning. I think from our experience, we found any excuse to get folks together to talk about teaching and learning, especially if it’s across disciplines, is always fruitful.

Tom: …particularly if there’s food. [LAUGHTER]

John: Yes. The third edition list you as author with Todd Zakrajsek, could you tell us how this collaboration came about and how it worked?

Tom: I think this was, in part, an insurance policy for the publishers. Given my advanced age, they wanted to make sure that they had someone involved in the project, who could take over if I just kicked the bucket. So, in the last six months of the book project, after I had finished writing the manuscript, Todd Zakrajsek joined with me and was really invaluable in finishing the process, in coming up with great ideas and suggesting great ideas, especially for the use of technology and AI with CATs, and in helping me manage the process with the publisher, which is much more complicated than it was 30 years ago. He also wrote the discussion questions for use by teachers and faculty developers, and I think those are going to be very helpful to folks.

John: We’re recording this in May, and the book is scheduled for release in June. We always end by asking, what’s next?

Tom: Well, I’ll answer that. But I’ll first say to teachers, what I always say, and that is, “Feel free to ask your librarian to order the book, so that you can use it, and your colleagues can use it.” Don’t tell Wiley and Jossey Bass that I said that, but I think that’s important. So what’s next for me is I’ve been working for a while with a group of people in, strangely enough, pharmacy education. I’m not a pharmacist, and I’m doing research with graduate students on what we call core concepts in developing what are the core concepts in pharmacotherapy? And that’s been very interesting for me. I’ve learned a lot from that. And so I’m ongoing with that. And we’re beginning to work on educational materials related to that. So that’s one of the things that I’m doing. And I’ve actually, I don’t want to commit to this too much, but I’m thinking about writing a book based on an article that I wrote that people have found useful, called “A Teacher’s Dozen.” And so I’ve updated that. I’ve been working on that actually for a few years, and something may come of that.

Rebecca: Well, we’ll be excited to find out what that is soon, maybe. But you haven’t committed, but maybe. [LAUGHTER]

John: Well, thank you. It’s been great talking to you. And my copy of the book is on preorder. And I am looking forward to receiving that.

Tom: So am I. [LAUGHTER]

Rebecca: Well, thank you so much for joining us. It was great to hear about the origin story of the book and how it’s evolved over time and I know a whole other generation of teachers will appreciate the new edition as well as teachers who have been practicing for a long time.

Tom: I hope it will be useful. Listen, it’s been a pleasure and a privilege to talk with you both. Thank you for giving me this opportunity

John: We should note that this is the first of two podcast episodes discussing this book. In next week’s podcast we’ll be joined by Todd Zakrajsek, who will be discussing his role in this work.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

339. Industry to Faculty

Some faculty begin teaching as a second career, after working in industry. In this episode, Kevin McCullen and Michael Walters join us to discuss how their prior careers in industry helped prepare them to design authentic learning experiences for their students.

Kevin is an Associate Professor of Computer Science at SUNY Plattsburgh. Prior to joining the computer science department at Plattsburgh, Kevin worked for several years at IBM. Michael is an Associate Professor and Chair of the Physics Department at SUNY Plattsburgh. Prior to joining the Physics faculty, Michael was the CEO of EISWorks Technologies and a metrology engineer for Corning Inc.

Show Notes

  • Design Automation Conference (DAC)
  • Consortium for Computing Sciences in Colleges (CCSCNE)
  • Clark, D., & Talbert, R. (2023). Grading for growth: A guide to alternative grading practices that promote authentic learning and student engagement in higher education. Taylor & Francis.
  • Lang, J. M. (2008). On course: A week-by-week guide to your first semester of college teaching. Harvard University Press.
  • CircuitPython
  • R.P. Colwell, At random – Employee Performance Reviews, IEEE Computer, 35(9), 12-15, Sept 2002

Transcript

John: Some faculty begin teaching as a second career, after working in industry. In this episode, we discuss how prior careers can prepare faculty to design authentic learning experiences for their students.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Kevin McCullen and Michael Walters. Kevin is an Associate Professor of Computer Science at SUNY Plattsburgh. Prior to joining the computer science department at Plattsburgh, Kevin worked for several years at IBM. Michael is an Associate Professor and Chair of the Physics Department at SUNY Plattsburgh. Prior to joining the Physics faculty, Michael was the CEO of EISWorks Technologies and a metrology engineer for Corning Inc. Welcome Kevin and Mike.

Mike: Thank you.

Kevin: Hello. I will just say 32 years is several.

John: It is. [LAUGHTER]

Rebecca: I think that’s the official definition. Well, today’s teas are:… Kevin, are you drinking any tea?

Kevin: I have Harney and Son’s Hot Cinnamon Sunset.

Rebecca: Oh, that’s a good one.

Kevin: It’s my go to tea.

Mike: And I accidentally left my tea in the car, but I do have a cinnamon tea that is by Tazo.

Rebecca: Yum..

John: Very nice.

Rebecca: Cinnamon’s are a favorite at my house.

John: I have that too, and I enjoy that one. Today, though, I have a spring cherry green tea.

Rebecca: And today I have Hunan Noir, which is very tasty.

John: Each of you received a PhD and worked in industry for a number of years before joining the professoriate. We’ve invited you here to discuss the transition from industry to academia. Many of us just went to school and never left. And we thought it would be interesting to hear the perspective of people who’ve been out there working and returned to academia. Could you tell us a little bit about your initial careers in industry?

Kevin: Well, actually, I took the roundabout path in that I joined IBM after my bachelor’s degree and IBM paid for my graduate work through programs that they don’t have so much anymore. And I always wanted to teach. And when I graduated as an undergraduate many years ago, I was kind of burned out on school. And then it gets hard to leave after a while because it becomes very comfortable. But eventually I kind of felt like it was time. I was becoming I would say more uncomfortable in the situation I was in in industry and opportunity presented itself. I had actually been looking for a number of years for a good local opportunity because I didn’t want to relocate. My wife still works for IBM. And Plattsburgh had an opening and I applied for it. And there’s a very funny story in there with the conversations with my management over that. But I applied for it, and they accepted me, and it’s been a lot of fun.

John: You mentioned a funny story, I think [LAUGHTER] that requires a follow up.

Kevin: Oh, it was just that I had submitted a presentation to an industry conference. So I’d gone through all the clearance work to get it approved, patent clearance and all those things, and it was not accepted. And then when the position opened up, and I applied for it, and Plattsburgh invited me to come over and give a presentation, I went to my manager, and I said, “So do you remember that presentation that we got approved for DAC? Can I go give it at SUNY Plattsburgh?” And she said, “Why?” And I said, “Well, they have an opening, and I’d like to apply for it.” And I said, “Don’t worry too much about it, because my guess is that they can’t afford me, even if they do give me an offer.” And she said, “Sure, go ahead and do it.” So I came back to her about a month later. And I said, “Good news or bad news. The bad news is they can’t afford me. The good news is I’m going to take the job anyway.” And we parted on good terms.

Mike: So I did get my PhD before starting in industry. And I sort of backed my way into industry. I was thinking about staying in teaching, but I was worried I wasn’t a good enough physicist to actually go for it. I have impostor syndrome very badly. And so I looked at going into industry because I was worried I was going to only teach freshman physics for the rest of my life, because that’s the only thing anybody would trust me to do. I went into industry for nine years, I did not have to approach anybody to see if it was time to leave, because they approached me and said, “Congratulations, you work your way into a research position at a production facility. That’s a pretty awesome job. And now we’re in a recession. So we don’t need you. Goodbye.” [LAUGHTER] I applied to a couple places in industry, had a position semi-offered to me at a solar panel facility in Oregon. And I could not say to the person who’s trying to hire, “Do you want this job?” He asked me three times. And after the third time, his shoulders just slumped and said, “You can just tell, okay, that was done.” And on my way home, my wife asked me, “Well, how did it go?” I’m like, “ah, I didn’t get that job.” I told her what happened. She’s like, “That’s lovely, dear. You have to figure out what you want to be when you grow up because we need a paycheck.” And at that time, I said, “Do you mind if I tried to go back into teaching? I think I really want to go back to it. Maybe I can not teach just freshman physics.” And so I adjuncted for a year at two institutions near my parents’ house. And SUNY Plattsburgh was looking for somebody who had a physics background but had engineering experience for their engineering three plus two program. I was a visiting professor for a year and then worked my way into a regular position.

Rebecca: I think there’s a lot of folks that are considering moving around and shifting careers at different points in their career. Can you talk a little bit about what that actual transition felt like that first year, maybe, back in academia?

Mike:Well, I can say three things that strike me when I was moving back into academia. Number one, I surprised myself finding out I can teach something besides freshman physics, so that was a great thing to figure out in getting some confidence there. The second thing was, and Kevin and I have joked around about this before, the pressures of academia in general, compared to the pressures in industry are at different scales. [LAUGHTER] And so we used to have fun when people would come up and talk about the pressures of doing stuff, we’d sort of chuckle a little bit, and go,”Yeah, this is pressure,” and I respect it, because there is definitely pressures in academia. But we can also talk about the times when you’re sitting around the office going, “Am I on the cutting block this time around?” …because you know, layoffs are coming this month, or even if you’re not on the cutting block, looking around at your friends and going, “Who’s not going to be here next month?” …is not a pleasant feeling. So being in academia made me feel more relaxed. I didn’t have tenure yet, but just the idea of knowing you belong, and you would know, hopefully, coming up if something was going to happen that way. And lastly, the camaraderie of the department … I am blessed to be in a department that was a very close-knit department. They were very supporting, they felt bad that initially, my office was in the middle of nowhere in a closet, basically, because that’s all the space they had. Kevin, I was on the fourth floor in one of those little side cubby holes. I only could see the sky from a skylight, literally everything else was cement.

Kevin: I thought you were talking about the office I first saw you in which also resembled a closet.

Mike: Yes, well, that was a different closet, [LAUGHTER] but it was a better closet. But anyhow, it was great having that camaraderie and having that trust, and that trust grew into a new major, because they trusted me to go ahead and take out the robotics major, and kick the wheels and let it go out for a spin.

Kevin: For me, it’s autonomy. I see a lot of people, I read the Chronicle and such, and it’s like making the transition to industry, and I’m thinking they’re going to have an interesting end of the year when they sit down with their manager to talk about how they did against their goals for that year, because it can be a little head spinning. My first year, I kind of felt a little lost, because it’s like if you had a guide, if everything you do is coming down from above, even if we get to a relatively high level in an organization like IBM, you’re driven by strategies that are sent to you and you are a piece of that strategy. And now you’re kind of an independent agent. And it took me a little while to get comfortable with that. But Mike and I both know people who really, really are struggling with the pressures of academia and all the things. I feel like I have a lot more ability to say “JNo,” and to say, “No, this is my limit of how much I am willing to take on and so I’m just going to steady on here.

John: I was talking to one of my advisees, who’s planning to go into a PhD program. And he said he was researching the market for a PhD economist, and he noted that jobs in industry paid a lot more than jobs in academia, and jobs in the government was somewhere in between academic jobs, and that, and he asked why there was such a big difference. And I used basically the argument that you just used, that you get a lot more autonomy when you’re in an academic setting, in terms of research, in terms of what you teach, in terms of what you focus on. And that’s something that I think we all value quite a bit. Was there anything that was surprising when you made the transition to an academic environment full time?

Mike: I’ll go with the autonomy. I mean, that was really surprising. Where all of a sudden nobody was breathing down my back. Where are the deliverables that you were supposed to get here? Other than getting classes together and doing your best that way? Yeah.

Kevin: I’m gonna go back to actually, it was a funny question I asked the Dean when I was hired, because I have kind of a long commute, as does Mike. And I was a little concerned about doing a commute from Vermont to New York five days a week on the ferry. And I asked the Dean, what are the expectations in terms of like, where you work, because even at IBM into the 2000s, we did a lot of working from home, pretty flexible work environment. And he said, one of the biggest changes when he moved out of teaching into administration was an expectation that he was in his office. And it’s the freedom and the flexibility. I’ll say one other thing, that was kind of weird, though. It’s the summers, and I’m a volunteer with several organizations. And immediately there was an assumption that I had nothing to do during the summer, and that I would immediately become available to them. And what I’ve taken to telling people is, the way that summers work is, I have a lot of things to do, but nowhere I have to be. And by the way, biggest surprise is that from what I have observed, we have the whole summer and everyone still does their syllabus three days before the semester starts.

Rebecca: We’re deadline driven, [LAUGHTER] just like our students. We are no different and that’s for sure. Can you talk a little bit about how your prior work experience has shaped your teaching practices.

Mike: So when I first came back into teaching, I taught in a very traditional sense. I would have the PowerPoint slides, I would have the graded homework, I would have the tests and exams in class so we can make sure everything is all above board and all that other fun stuff. And then as I go through, I think more and more back on what was important when I was working, and how that work environment operated. I now have moved as far away from trying to test as minimally as possible, if not never, and do much more project-based learning because that’s what I did. And even when I give a test, I would give a two-fold piece of the test. The first bullet piece is the closed book portion, which never ever had a calculation in it. And I told my students, when I was at a meeting, I was there as a knowledge source. And so I’d have to know somethings off the top of my head. That’s what they paid me the money for . And then if they asked me a question about X, Y, or Z, that had to have some calculations based into it, we’ve never had to do it right there on the table. That was not something you did. You could go back to your office, you could use all your resources, you could pull together the things you needed for that project. And so I tried now to model my classroom along those points, I will tell them, these are the things you need to know off the top of your head, this is the stuff that they’re going to pay you for that boom, boom, boom, you need to know. But everything else, Google is your friend, having your textbook there is your friend. And so on any of that stuff it’s open book, open notes. It’s a little harder with AI now. I’m not going to lie. But still, in general, if I was in industry, that would probably be where I would start as well. Oh, I need to do this. Okay, “Hey Google blah, blah, blah, blah, blah.” There’s my starting point. Now let’s move on forward.

Kevin: I think two things. So I spent the 32 years that I was at IBM, I spent one year doing microprocessor design and the other 31 years, I was in software development. And I’m a big believer in soft skills. And some of that comes from also being an ex-Scoutmaster, and heavily involved in scouting. But I tell my students, I try to instill in them the idea that your communication skills are vital to your advancement in your career. Everybody can code, but as you advance in your career, as you go further up the ladder, you’ll have larger responsibilities and you have more communication responsibilities, till you reach a certain point in which you live from being as technical toward being more in terms of communication. But one of the last things I worked on was I was basically project managing the, what they call the enablement for a test site, which is to say, just getting all the pieces together to build an experimental chip and a new technology. And that was primarily a communication job, it was a project management kind of thing. The other thing is the importance of group work and project work. And being able to work with other people that those, I think, as well as for my computer science students, algorithms really do matter. And anybody can write code that slow. But it’s important that you understand enough of the algorithms. And that’s because I entered industry as an electrical engineer, I had no background in any of those things. And I wrote some spectacularly bad code, when I was back in my 20s, before I started taking some graduate courses and going, “Oh, wow, there’s a better way to do this.” And so those are kind of the things that I think are important.

Mike: I want to piggyback on the group work part, I do a lot of group work as well. And one of the things I tell my students because I will always get the “Well, I’m the strongest person on this team, I’m doing all the work.” And I go to a “Yeah, congratulations. I said there was never a project that was less than $20,000 I worked on by myself. You’re always part of a team, you’re never a loan problem solver, if there’s money involved, because they want the others to be part of it and keep track of it and make sure it happens as fast as possible.” So group work where sometimes we let them off because of the whole stigma of “I’m going to be the only person working on it. Everybody is going to be a hanger on.” Well, when you get paid, that’s going to happen too and all you have to do is smile and say, “Yay, thank you,” and keep going [LAUGHTER] because you’re getting paid to do the thing.

Kevin: But I’ll just add that there’s no industry group I’ve ever worked on, where there weren’t checkpoints and communication of where things were. And so I know Mike does a lot of these kinds of things as well, which is the idea that what happened when we were undergrads where they basically say, “alright, you’re in a group, go do this, and then deliver the product.” So even in student groups, the importance of having checkpoints, and I basically have them write progress reports like you would do in industry along the way. It’s not “Here’s the project, go do it. Now here’s an end result.” It’s more like “Okay, you’ve gone for a week. What has each person done? What’s your next step?” Those kinds of things.

Mike: And while we’re trying to develop these skills for students going into industry, we think that there… well, I say, and probably Kevin will agree, they’re important no matter where you’re going to go into, including academia or any of the other spots. So I don’t want to make it sound like we’re trying to educate students just to go into a particular engineering job. But this is more of a realistic facet of life in general.

Rebecca: Do you think you both would have found this same path in teaching had you not had the industry experience and kind of underscoring and value some of the same things that you do now?

Mike: Absolutely not. You like what you’ve seen, and so we would have seen “sage on the stage,” and that is what we’d do. And to a point, that’s where we all start, right? I mean, there’s that sage on the stage piece, and then you start saying, “Okay, there’s got to be something better, because I’m not reaching all my students.” So how can I do that with the knowledge that you never will reach all of your students, but you got to give it the best bet you can.

Kevin: I think when I started, one of the big differences was the deference that people showed, it’s like, “Oh, you’re a faculty member, oh…”, and I worry, actually, if I had been 28, and started teaching at that point, that that would have gone to my head, and I would have started to believe my own PR. I don’t see people around me with that problem, but kind of a stereotype of academia. And you do read stories about actual faculty who are like that. But working in industry can give you a good strong sense of humility, because you have successes, and you have failures along the way.

Mike: So I will say in industry, there was still some of that out there. I mean, I used to get emails from my PhD colleagues down in Painted post, who would talk down to me because I was at a plant saying, “Oh, let me explain this to you in very simple terms.” In the very next email, I would just respond. “Well, thank you so much for your guided steps. Dr. Michael Walters, PhD da da da da da.” it was amazing. The next email was much more as an equal instead of I will talk down to you because you are way out there in the hinterlands. Do you know how to rub two sticks together to get fired, because you might need that you’re in the cold part of the country.

John: That reminds me of a time when I was coaching soccer for my sons’ teams. And there was another child who was on the same team three or four times over a space of four or five years. And there was an article that mentioned me in a local newspaper, and this mother came up to me and said, “Well, I knew you worked at the college, but I assumed you were a janitor there.” It was an interesting experience, because she always saw me out there in sweatpants and t-shirts and just assumed “Well, that’s not how a professor looks.” I think maybe that attitude is more common in elite private institutions, Ivy League, and so forth. I don’t think I’ve seen it at Oswego, which is very much like Plattsburgh, in terms of being a four-year comprehensive college.

Kevin: Well, if you wanted to, you could have coached in the tweed jacket with the elbow patches, and then they’d have all known.

Mike: And a pipe, you gotta have the pipe. You don’t have to put anything in the pipe, because that’s bad. But just the pipe itself.

John: I actually had one of those sports jackets. And it was a running joke with some of my students back in the 1980s, [LAUGHTER] because I just had that one sports jacket, and I wore it whenever there were honors, awards, and so forth. And they still ask me about it 35 years later. [LAUGHTER] But you know, one of the things I’m hearing from both of you is that you’re both doing some authentic assessments, which is something that we’re trying to encourage faculty to do in general, to move beyond the traditional types of teaching. How do you find that students react to these types of group projects or the teamwork or other things that you’re having them do?

Kevin: I’ve had, I think, a lot of success with it. I can’t think of any really bad experiences. Mike’s nodding his head. But I have a poster actually coming up at CCSCNE on our tech startup class, that actually, Dr. Del Hart, our department chair, the person who kind of came up with this idea, and then he came to me and he said, “So you’ve worked in industry, and you’ve been a Scoutmaster, how would you like to try leading this class, and it’s a multi-level multi-semester class where students basically can take on different roles based on whether they’re 400, 300, or 200 level in the class, and projects can run for multiple semesters?” And it’s probably one of the most fun classes I’ve ever taught. Because a lot of times I found myself, I don’t teach it now, but I found myself actually telling them “Slow down a little bit. Let’s establish some realistic goals. I know you’re very excited about what you want to do here.” But I think generally, I find the project work goes really well. But I do try really hard to make sure that I’m aware of what they think is going on with their team through various assignments that I collect that are like reflections or statuses.

Mike: I’ve had overall success. There have been times where there’s been failures, one or two semi spectacularly. We do a intermediate robotics lab, which is a project-based group class and at the beginning of the semester I give them what I call a 90% google-able project and say “Okay, from here on out, your group is now doing this for the rest of the semester. If you get done with part A, there is a part B you can have as a stretch goal if you want, but this is what you need to do.” And I try to emphasize that failure still means you can get an A, it just means that you failed at trying to get the project to work. The 10% is the hardest part of the whole project for any of these projects. Working for Corning, I was working on the bleeding edge of technology. And so I was always a horrible person till then in saying how long it’s gonna take to do something because I’ll say “it takes two weeks,” because it looks simple. And at the end of the two weeks, my boss would come back to me, “How’s that going?”” I got the first step kind of done.”” What do you mean kind of done?” “We’re still inventing something brand new, it didn’t happen as fast as I was hoping.” So when the project does work, this 90% thing, it’s great to see the students get excited, they see progress, they keep moving. And then what’s funny is then the cockiness will come out and say we can get this done in that three weeks, I’m like, “Yep, you’re just like me, we’ll see you at the end of the semester.” And sure enough, [LAUGHTER] that’s when it’ll happen, because they’ll run into wall after wall after wall because that last 10% is hard. But I do get sometimes, like this past semester, where the project I picked ended up being a failure. By the end of the semester, it did not work the way it was supposed to work, they could not get the robot to do the thing. And you can tell that towards the end, they were getting really down. I mean, really, really down. And at our final session where we basically talked about what worked and what didn’t work for the project, basically, I told them, “This is what happens probably 25 to 50% of the time in industry. You’re going to work on a project, work on a project, and it’s not going to work.” And there’s times when you got to sit down and say why doesn’t this work? And then go to your boss and say, “Well, you either a didn’t give us enough information, or enough resources or a combination of the two.” And I told them, “I didn’t give them enough resources to make it happen.” But we got to this point. So I pumped them up like, “Here’s the teamwork that you did, this is all great.” And they all got their “A” because every single person contributed and worked towards it. But those failures can sometimes be problematic. On the flip side of it, doing so much non-assessing means I go ahead and give them a lot of homework assignments that once again, are graded basically, “did you hand it in?” Did it kind of do the thing was supposed to do? Yay, you get all the points. And what’s funny is I thought this semester where I cut up all my tests. I have no test this semester whatsoever. Everything is you have homework, classwork, and a project. That’s it. And the beginning of semester, I was nervous. I was like, how am I going to differentiate? Not that I care. If all the students get As I’m happy. But I just want to know if I had to differentiate, how am I going to do the differentiation based on the system? Well, my students have proven to me that they can self select, because they stopped doing assignments. So they stopped doing whatever. And it’s like, “okay, this is how self selection will work.” It’s not that I don’t even come to them, say, “by the way, you haven’t handled this in…” “yeah, I’ll get to it.” We’re human, they’re not going to get to it. And that’s fine, too. But it ends up being almost more psychological of “You didn’t prove to me, not only that you didn’t know the material, but you also showed that you didn’t have the discipline to get that piece done.” And whether that is the focus of my class or not, I don’t know. That’s the negative to this type of assessment. I can’t tell if it’s because they didn’t know the material. So therefore, they didn’t feel comfortable trying to finish the assignment, or whether life happened. And that’s the only negative I’ve seen so far about using this assessment, because you see them not be so stressed, because there’s a test coming up and not be so focused “I’ve got to cram this thing together,” then five minutes later don’t know a single thing when you go talk to me in the hallway about what I put on the piece of paper.

Kevin: It’s funny in that class that I started by actually did something that would be called ungrading. And I used something I’d found in an article years ago, I could find the reference if I had to, but it was how Intel assessed people is managers would come in initially with a sheaf of overhead slides, then they’d have “this person should be promoted.” And you look at it’s here slide 1 of 60 kind of thing. And they went to this thing, I think they call it three up, three down, where you had to give three positive and three negative for the person and then summarize. And I actually had my students in that class for their midterm and their final grade, they had to do those. They had to do a self assessment, they had to tell me three things they were proud of, three things they could have done better or they didn’t do well, what they thought their final grade shouldn’t be and why. That class, though, that was a pretty much a highly selective class, that it was very, very seldom that a student checked out. And part of that was because the person in charge of their project was the 400 level student who they had agreed to work on that project with. And so there was a lot of peer interaction that led to just generally, I think, really good outcomes. Dr. Lecky’s teaching the class now, and from what I hear, he’s having a very similar experience with that course sequence.

John: One of the things we’re doing right now, and I know Kevin, you’ve already been participating in this, is a reading group on Grading for Growth by Robert Talbert and David Clark. And one of the points they make in their book is that, while we’re so used to traditional grading because we grew up with it, and that’s what we’ve always done in some way, one issue is the only time people tend to be graded this way is in academia, when they’re working in industry, they’re reviewed, but they don’t get grades, they either meet the requirements or they don’t, or if they exceed them, there are chances of promotion and so forth. But in pretty much everything else, including our own careers, we get renewed, we get tenure at some point, and there may be some other additional pay increases based on how well you’ve done. But we don’t get letter grades. And we don’t get numeric grades every year. And one of the things I’m noticing is you’re both doing some types of grading that are closer to the types of experience that you’d find in industry. Do you think that made you perhaps more willing to try these types of things that a lot of faculty have been really reluctant to experiment with?

Mike: I think so. When we talk about the industry thing it brought back a story, I one time was in a review, and I was rather well paid at this plant. And so they had high expectations for the salary they were paying me. I remember, I was talking to the plant managing engineer and they even brought the plant manager in during my review. So either this was gonna be a really good thing or a really bad thing. When I walk into the room. It’s like, okay, which way is this gonna go? I thought it was a bad thing. But what’s funny was they gave me a “met expectations.” But the plant manager turned to me he and he goes “I expected you to “exceed expectations.” I’m like, “Well, how am I supposed to exceed expectations? By default, you’re saying that is your expectation. So therefore, I either met the expectation or I didn’t.” And we got into a heated verbal argument for a little while. And eventually, I said, “Look, you either give me met expectations, or I didn’t meet expectations, I don’t care. But what you just said makes no sense.” And she threw up her hands and left, and I got “met expectations.” But it was just one of those things. It’s like even using these types of methodologies, sometimes you get conflicting viewpoints that goes forward. As to what your question asks, yeah, I think with the industry, this is why I’m gravitated towards that. I teach a class called fundamentals of engineering design. And when I tell the students why this class is there, I said, it’s everything I wish I knew as an engineer, before I became one. I was a physicist, I didn’t go through any of these preparatory courses, and then boom, here’s an engineering role, go run with it. And so being able to help my students be in a better situation when they go out into industry, which I assume most of my students are going to do, is a good thing. And any of the skills they pick up because of this isn’t bad if they stay in academia. Maybe this helps break the cycle of, we only can test and do homework for grades. By the way, I’m just gonna throw out my pet peeve for all of you out there listening. My biggest pet peeve is if you grade homework like a test, and count it, where are the students supposed to make the mistakes? That’s the thing that always used to tick me off. Where do I practice and get legitimate feedback on my practice, to know where I’m making a mistake such that when I go ahead and then perform, I know I’m performing at a higher level. That part always bothered me when I was an undergraduate. That was from day one, when I started adjuncting, I said, I’m not going to do that. Homework is going to be: did you try it or did you not try it, period, That’s it.

Kevin: I’m gonna make just one little thing. IBM really, really tried very hard for a long time to just grade people. We had a system where you had an appraisal or a personal business commitment. And at the end of the year, your manager basically gave you a score from one to five. The system changed many times over my years, but: one was, you’re on track, we’re going to promote you, you’re doing awesome, and a five was it was nice knowing you, but you generally knew where you were going to fall. And I won’t get into all of the arguments over the years as to whether there were ratios or stacked ranking, and at what level in the organization, there was a quota on how many could be at each thing. Actually, those kinds of systems caused a lot of the negativity that made me think that maybe I wanted to go to teach. I think, for me, one of the really big things has been, as soon as I started teaching, I started trying to learn how to teach, because I walked in the door, I adjuncted for one class at the University of Vermont, it was a graduate-level course and I really enjoyed it, and it was great. But I realized how much I didn’t know about teaching. I had my teaching statement when I applied for the job, actually leaned really, really heavily on stuff I’d learned as a boy scout leader, and then like a leadership course called wood badge, a six day leadership program. And I had learned a lot of what was basically active learning as part of that program, something called the edge method: Explain, Demonstrate, Guide, Enable sort of thing. And so I had kind of this foundation that came from a very non-traditional place in terms of pedagogy. And I came in and I started out doing exactly what I had experienced, which was you stand up, you lecture, you use PowerPoint, it’s really cool, you give homeworks but at the same time I started reading stuff and a lot of it was James Lang’s books. I think I’ve read all of his books, because the first one I read was On Course, and I think that they should give every new faculty member a copy of that book during orientation, because it got me through my first year. I have a copy in my office that I think has got a thousand post it notes stuck in it. And then I started reading his other books. And then I started reading the books he was talking about. And these days, I love to teach my classes in our programming lab, because when I can, it takes more prep time, but it makes actually the class easier for me in a lot of ways to do a little bit of lecturing and then say, “Okay, now you get the computer in front of you, go make JavaScript to do this.” And I’m going to walk around the room and help you do it. And so you’re kind of turning even lecture sections into labs, in that sense. And I find that’s a lot of fun for me. And I think the students find it more interesting, because some of them kind of struggle. But the thing is, there’s me, and there’s the other students around them. Like when I give quizzes, now I hand out a quiz, I say, “Take about 10 minutes and work on this.” And I say, “Okay, now compare your notes with your neighbors.” And then I say, “alright, let’s discuss it.” And at the end, I collect it, and basically say, “There you go, you get credit for being here, because you did this.” And I have a couple of students and Mike would know their names, I won’t mention them, they sit there, and when I give him one of these things, they’re way beyond anything I’ve even envisioned, they’ve gone off. But other students, I have to sit with them and help them. And I just find that’s a lot more satisfying for me as a teacher than to stand at a whiteboard and scribble on it for 50 minutes, or mine are all an hour and 15 minutes this way I set my schedule up.

Mike: I’ll agree with that, and I’m actually moving more and more of my courses over to, I will give you the bare minimum that you need to do what I’m about to ask you to do, and then go do this thing. And practice it there. And I’ll walk around, I’ll help out, we’ll talk to everybody, make it much more of a laboratory class, even though it’s not a laboratory class, just because the interaction between their peers, their interaction with me, becomes more of a one on one instead of one on many. Way back when, I used to start off my class, when I would teach introductory physics with saying, “You could learn from the book everything I’m about to show you, and that’d be great. And if you never show up and just show up for the test, you could do that. Awesome, congratulations, we’ll see these dates and have a great time. I know. But if you come here and just listen to me, that gives you the book and me. But here’s the more important thing, look around the room, there’s 40 other people here. So that’s me, 39 other people and you. All of a sudden you have lots of teachers that could work with you. And as long as I help facilitate you being able to use that resource, you have many, many more options, because I might not have the analogy you need. But your friend next to you or the one over two rows from you might have that analogy you need to make that connection.”

John: Could you each share some of your favorite teaching activities that you use in your classes?

Mike: I’m trying to teach them brainstorming methods, especially technical brainstorming. I have 1 2 3 4 5 6 whiteboards in my room. And I’m going to ask in front of each whiteboard, “I want three technologies, just random three technologies.” I’ll keep writing them down without telling them what’s happening, three technologies on every whiteboard. And then I break them up into groups and give them each one color markers. They always use their color marker and say, “Okay, now the six groups go to each of your boards, you now have two minutes to write every single technology product you can come up with, that’s a combination of some sort, matter, or form of those three technologies.” And sometimes like it could be weird stuff like ice cream, rocketry, and such and such, and I’m like, “Yep, now you got to figure out how you’re going to combine those three.” And I tell them, “Okay, so if you’re the first person on the board, you’re gonna put a one next to your thing, you only get one point for every one you come up with. Then you’re gonna draw a line and the next group comes up, they get two points per thing they come up with, they can’t come up with something that you already have, all the way up to that final group, where now five of the groups that brainstormed, but everything you come up with now is worth six points.” So it’s like you get one or two things you just made up for your first board and second board without a problem. But it really shows them, sometimes those six pointers at the end, when you’re really just grasping at straws, become the things that you want to move forward on. It goes to writing even. So I know when I do my writing, it’s like that first draft, you did all the easy stuff. Now throw out all the easy stuff, and I’ll start putting the second and third order stuff that might make it much more interesting.

Rebecca: That’s a great exercise. It’s really hard to get students to get to that further step. They want to stop at step one or two.

Mike: Oh exactly, and making it a competition of which team can get the most points, get some of them, their juices flowing. And it’ll be funny because at the end, we go through everybody’s stuff like “is this a legitimate idea?” No, cross it off. Then it doesn’t count. That’s a group response of rating, basically, of the thing. And it’s funny because at the end, most of the time, I can’t get through all six boards by the end, and they’ll be sitting there going: “We got to finish this. I want to know who won.”

Kevin: …favorite teaching activities. I really do like and I do it a lot now, which is to give the students little coding challenges in C++ or JavaScript class, which is to basically hand out something that looks like a quiz, but it’s basically: here’s some coding, let’s do do this simple coding thing in class, because if I handed out as a homework, well, they could try using ChatGPT. Although I have to at some point talk with Mike about the fact that one of my students tried that and ChatGPT hallucinated a library for circuit Python that doesn’t exist. And as I said, on social media to my friends, I’m happy to set up a PayPal account, if ChatGPT wants to hire me as a consultant to debug their program’s mistakes, because the poor student had no idea what to do, because the code he was given was incomprehensible. But I really like giving the students an exercise to do in class. The second place I would say, was, when I taught our ethics course, I did a lot of debates in class. And sometimes I would let them self select what side they wanted to be on the issue like drone warfare things, other times, I would kind of assign them. And when they balk at that, I’d kind of lean hard into my own experience and say, “Well, you know, I debated in high school, and you didn’t get to choose what side you were on. And it helps sharpen your brain to argue something that you don’t necessarily agree with, to try to logically follow it.” I would say for in classroom, those would be for out of classroom, like project kind of things, the posters, having students do a poster session, at the end of the semester, actually printing the posters, and putting them up in the lobby of our building. And then having kind of a little session where people can walk around, just like a conference poster session.

Rebecca: it’s really great to hear the different ways that both of you have incorporated your industry experience into the classroom from changing grading systems to thinking about consistent feedback throughout the semester so that students can learn and always being on that verge of not quite knowing what’s coming next, like you are often when you’re in a tech field and having to constantly learn or have professional development or try things that are brand new to you. But we always wrap up by asking: “what’s next?”

Kevin: I am, as the phrase goes, playing with house money. I was at IBM for 32 years, my wife still works there. I continue to do this because I really enjoy it. And I love all three aspects of the job in academia, I love the research, which we do research, I think I would say with a small r,, not like R1 with a capital R. I love the research, I really liked the service. All that time I did at IBM, I can do committees like you wouldn’t believe. I know how to do that kind of stuff. And I know how to be helpful in that way. And I love the teaching. I mean, I’m teaching an honors seminar on algorithmic bias this fall that I’ve taught once before. I really enjoy teaching a new class every once in a while or creating a class. It’s a very intellectually stimulating field. And I’ll just do it until my wife and I decide it’s time to be able to go to Disney World in September or something. And my guess is that a few years down the road, I’ll probably retire. And I will continue to adjunct because the great thing about adjuncting is you can keep an email address and you can also still get discounts on software. [LAUGHTER] I kid you not. It’s kind of like “Hey, I’ve got all of the Adobe tools and I was playing with Dreamweaver for my JavaScript class and generative AI creating images the other day.” I’m so curiosity driven that continuing to teach as long as I’m enjoying it.

Mike: My “what’s next” is to try to become a better evangelizer for the robotics major here at Plattsburgh. We’ve been growing it slowly, and I firmly believe in what opportunities it allows people to have it going towards the future. I run a not for profit here in Plattsburgh that works with middle schoolers and high schoolers to give them robotics opportunities on Saturdays. I do this major, I’m chair just so I can make sure I have the time to devote to doing things. It is fun, starting something and watching it grow up. And I’m just having a ball with it. All the stresses and anxiety that come with that as well. But yeah, that’s my what’s next.

John: And going back to Kevin’s comment, before you make the decision to go to Disney World in September. If you start teaching online or doing some more work teaching online, there is the Online Learning Consortium, which has a conference at Disney World in November, which is a really great time to leave the Northeast. And it’s a four-day conference, but you get the special conference rates at the hotel for a couple of days before and a couple of days after. So you might be able to combine those things.

Kevin: I will have to look into that because the best conference I ever went to is Design Automation Conference, it was held at the San Diego Convention Center and I went to that a couple times for IBM for CAD software and it’s one of those 6000-person trade show conferences, top end technical conference and on the trade show floor, everybody’s collecting polo shirts and desk-y gadgets, that kind of thing. It can be a lot of fun. And I have stories about that that are actually very funny as well, because when you’re at one of those and the part of the company you work for actually competes with many of the companies you’re visiting, eventually they kind of figure out that you’re asking questions that an ordinary customer would not ask. [LAUGHTER] And they’re kind of like, what exactly do you do for them?

Rebecca: Well, thank you so much for joining us. It was great to get a chance to talk to you in more depth.

Kevin: This was fun.

Mike: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

Ganesh: Editing assistance by Ganesh.

[MUSIC]

315. Unessays

An unessay assignment provides students an opportunity to demonstrate their learning in innovative and creative ways. In this episode, Jessamyn Neuhaus and Maggie Schmuhl join us to discuss how they have employed unessay assignments in their courses.

Jessamyn is the Director of the SUNY Plattsburgh Center for Teaching Excellence and a Professor in the History Department at SUNY Plattsburgh. She is the author of Geeky Pedagogy: a Guide for Intellectuals, Introverts, and Nerds Who Want to be Effective Teachers. She is the editor of Picture a Professor: Interrupting Biases about Faculty and Increasing Student Learning. Maggie is an Associate Professor of Criminal Justice and the Associate Director of the Center for Excellence in Learning and Teaching at SUNY-Oswego.

Show Notes

  • Neuhaus, J. (2019). Geeky Pedagogy: A guide for intellectuals, introverts, and nerds who want to be effective teachers. Morgantown, WV, USA: West Virginia University Press.
  • Neuhaus, J. (2022). Picture a Professor: Interrupting Biases about Faculty and Increasing Student Learning. West Virginia University Press.
  • Neuhaus, J. (editor) (2022). Using the Unessay to Teach History. Teaching History: A Journal of Methods. Vol. 47. No. 1.
  • Cavanagh, S. R. (2023). Mind over Monsters: Supporting Youth Mental Health with Compassionate Challenge. Beacon Press.
  • Jessamyn’s American Promise Unessay Assignment
  • Pittman, Chavella (2022). “Strategizing for Success: Women Faculty of Color Navigating Teaching Inequities in Higher Ed” in Picture a Professor: Interrupting Biases about Faculty and Increasing Student Learning. Ed. by Jessamyn Neuhaus. West Virginia University Press.
  • Pittman, Chavella (2023). Navigating Teaching Inequities. Tea for Teaching Podcast. Episode 291. May 31.
  • POD Network Conference
  • Teaching, Engaging, and Thriving in Higher Education Series. University of Oklahoma Press.
  • American Historical Association Annual Meetings

Transcript

John: An unessay assignment provides students an opportunity to demonstrate their learning in innovative and creative ways. In this episode, we explore ways in which different faculty have employed unessay assignments in their courses

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guests today are Jessamyn Neuhaus and Maggie Schmuhl. Jessamyn is the Director of the SUNY Plattsburgh Center for Teaching Excellence and a Professor in the History Department at SUNY Plattsburgh. She is the author of Geeky Pedagogy: a Guide for Intellectuals, Introverts, and Nerds Who Want to be Effective Teachers. She is the editor of Picture a Professor: Interrupting Biases about Faculty and Increasing Student Learning. Maggie is an Associate Professor of Criminal Justice and the Associate Director of the Center for Excellence in Learning and Teaching at SUNY-Oswego. Welcome back, Jessamyn and Maggie.

Maggie: Hello.

Jessamyn: Hi. Thanks for having me.

Maggie: Good to be here.

John: Our teas today are:… Jessamyn, are you drinking any tea?

Jessamyn: I am. In preparation for this very question, I’m drinking a wild sweet orange.

Maggie: Oooh.

Rebecca: Sounds nice.

Maggie: That sounds very fall. I am drinking a bread pudding tea. It’s a black tea from a little tea shop in Sacketts Harbor.

Rebecca: That’s a first on our podcast for sure.

Maggie: I think it’s called Tea Thyme, the little shop. It’s quite lovely.

Rebecca: Nice. I have Irish breakfast this morning. So we’ve invited you both here today to discuss unessay assignments. Could you tell us a bit about what an essay is?

Maggie: I guess I’ll say an unessay is anything but an essay. Jessamyn probably has some more nuance? [LAUGHTER]

Jessamyn: Not really, that is the basic premise. In classes where you might traditionally ask students to show their learning via a standard essay, either like the five-paragraph essay or a research paper, an unessay project asks students to demonstrate their learning, increased understanding, via any other kind of format.

Rebecca: Jessamyn, can you tell us a little bit about why you decided to use unessays in your class?

Jessamyn: Sure. And this also relates to… I know, we were maybe going to talk about what was appealing for historians, in particular, about the unessay. And some points for history professors generally and me, what attracted us to considering using the unessay. A couple of things about our discipline: first, that the unessay, it’s a good way for students to explore some key historical thinking skills in different new ways for them, so things like citation, research, and historiography. So trying to understand what other historians have written and argued about the past. There’s also been a pretty significant sort of public history turn in the world at large, like trying to better understand different audiences for historical research. Some other more general reasons were issues of equity and inclusion, especially around Universal Design for Learning, trying to offer students some options as a way to increase their learning, increase engagement. And the final thing I’ll say, just for me personally, is that I wanted to make assessment and grading a more joyful and happy part of my teaching life. It had really started to drag me down. And I didn’t want to approach assessing student learning as a soul sucking slog, not just for personal reasons, but also that it was interfering with my ability to accurately assess. If I was absolutely dreading reading a stack of essays, then something was amiss, and I wasn’t going to be able to, in my view, really accurately assess what has this student achieved.

Rebecca: About you, Maggie?

Maggie: I think really a lot of what Jessamyn said echoes in the reasons I decided to incorporate the unessay in my classroom. I was actually inspired by a talk that Jessamyn gave to our faculty, I guess it was over a year ago now, about the unessay and I immediately thought of my senior seminar class. So in criminal justice, in our department particularly, it was common for students to write a 20- or 30-page long research paper, and they dreaded it. It was a source of anxiety for students. It was something in the end that I’m not entirely certain they felt proud of. They just were trying really hard to get the A in the class, to make it perfect, to spend lots of late nights trying to perfect this project and I don’t think it brought about the kind of learning gains that I was hoping to see with my students, it seemed like there was a lot of going through the motions. And the unessay just gave us an opportunity to really turn that on its head and give some of that autonomy back to students and some of that creativity back to learning.

Rebecca: I’m loving these themes of joy, creativity…

John: …which are not terms we often associate with either writing papers as students or with grading them as faculty, and Jessamyn, you were the editor of a fall 2022 issue of Teaching History that focused on unessays. Could you tell us a little bit about ways in which you and other historians have been using them in your classes… some examples of ways in which students have demonstrated their knowledge?

Jessamyn: Sure, and I’ll just give a little promo here, Teaching History: A Journal of Methods is an open access journal. So you can just Google Teaching History: A Journal of Methods, and it’ll take you to the open access platform. It’s hosted by Ball State University. And so that issue is still totally available for anyone interested in reading it. I would say, like some specific projects that were explored in that issue, there was an article about using maps as historical documents, there were a number of projects that were exploring how to present history and historical thinking in a public history setting. So along the lines of museum exhibits, and especially in a digital format, the upsurge in digital humanities is something several of the authors were looking at. And then there’s a whole range of examples that were discussed in the two interviews I did for everything from board games, to graphic novels, to quilt, to the annotated recipes. All those were discussed in that issue.

Rebecca: What are some specific projects, Jessamyn, that you’ve done in your classes,

Jessamyn: A couple that have come up that I used… it’s interesting… Maggie used the unessay with her senior students. And I have helped students in the history department here, they also have to do a major research project. And I’ve been the advisor on a number of non-traditional projects is what I called them at the time, and now would say unessay. So at the senior levels, students did major research projects, like a fictional interview with a historical figure, historical fiction would be another example. I’ve had a couple of students do online exhibits and one person do an in-person exhibit. But I also use the unessay assignment in my intro level class, which is a basic history survey class. And those projects ranged from again, the board game, I had a number of posters, I had one student who was a trained professional baker do a set of cookies with icing, comparing different protest movements, the Bonus Army March in the 1930s with the more recent climate change activism. And one of my very favorites was a video recording that a student did, showing step by step how she made a family recipe. And since the theme of the class was the American Promise, and the way that throughout our history, as a nation, we’ve struggled to achieve the ideals of equality and liberty for all. And we get closer and we fall short. And her recipe was from her family’s background from India. And she took us step by step through it. She had music playing and was describing the importance of this particular dish in her own family history. The unessay was encouraging them to see their location in the American Promise. And it really struck me, in my previous life as a historian of food and food waste. So that was interesting. But most striking to me was, this was a student who had been very, very quiet in class. I felt like I didn’t know her very well. She was doing great work, but she was sort of on the edge of things it seemed like, and with this unessay, it completely changed my perception of her. I felt like I got to know her so much better and so did the other students as well. Having students share their unessays is an important part of the work and It really changed just the whole classroom dynamic in that way. So that was really notable.

Maggie: Nice.

John: Maggie, what are some of the projects that your students have done as unessay assignments?

Maggie: Yeah, so I mentioned before that I assigned the unessay for my senior seminar class. So it’s a sort of capstone project. And there were some really awesome projects that came out of this and I probably don’t have time to talk about all of them. So cut me off at some point when I’ve been going on for too long. But I had a student who wanted to be a elementary school teacher, and had gone through our program and decided that being a practitioner in the field of criminal justice was not exactly where they wanted to be. And so his project was to create lesson plans for second graders to communicate some of the harms of incarceration and how it affects youth when their parents or caregivers are incarcerated. And so it was a really wonderful project, all the students had worksheets, kind of like you can think about having when you were in second grade, and little matching puzzles and games, and it was a really cool experience. And for him getting interested in a master’s of education program, it was a way that he could then take that project and show to his grad school applications and advisors, like, “Hey, this is something that I’m really passionate about. I’ve been working towards this. And yeah, it was a lot of fun.” We had a good time, even some spelling words thrown in there. [LAUGHTER] It was great. Another student project was a choose your own adventure story. There’s various examples of this. But in my youth, it was the Goosebumps choose your own adventure where you read to a certain part in the story, and then it gives you the option of what you want your protagonist to choose. And then you have to flip to a different area of the book and see the fate of that character. And so the students started a really great project, unfortunately, didn’t get to finish it for various reasons, but started a really great project about reentry. And so the main character was someone who was being released from prison and had to navigate all of the barriers and circumstances around reentry that made it difficult for him to assimilate back into society. So that was a big fan favorite amongst the class. They really enjoyed that. Of course, we also had some board games, one in particular, was a monopoly game that examines varying socioeconomic status, and how that influences someone’s path through the criminal justice system. So the players started off with different amounts of money. And as they would pull community cards and have to purchase property and such, they would be challenged to need to pay a particular court fine, and how that would set them back from being able to find a job. It was a very well thought out game of Monopoly. But the goal to show and highlight the inequalities related to income and how that affects criminal justice processing. I could probably keep going, but it really gave a lot of opportunities for students to combine what they were interested in. There was a student who had run their own podcast for a period of time. So she conducted a podcast series on Amber Alerts and their effectiveness. And it was something that when she had been traveling down to Texas, and had received a number of Amber Alerts in the short time that she was there, she was like, “Are these things really working? And how can I explore that through a medium that she was already quite drawn to?”

Rebecca: Unessays always sound like so much fun?

Maggie: Yeah. And they’re fun for us. I know Jessamyn touched on that, too. It’s a whole different experience as someone assessing the work that they’re doing when you know that they’re passionate about it. And it gets me excited about assessment, which I don’t think is something we can typically say.

Rebecca: There’s not a big line of faculty raising flags saying assessment is fun.

Maggie: Right. [LAUGHTER]

Jessamyn: I also wanted to follow up on a point Maggie’s touched on a couple times about student autonomy and agency. I think for both our student populations, this is a really key academic, professional, and personal skill that many of our students need to build, that they arrive to college with a sense of themselves as very passive recipients of education and have not had the opportunity to really see themselves as producers of knowledge. as having agency in academic settings. So for me, one of the great glories of the unessay, is that it’s starting point really is, what do you want to do? What do you want to do to demonstrate that you understand this topic, and for a lot of students, it’s scary at first. A few might think that you’re trying to trick them. And I know this was something we did want to talk about. It does require, on our part, a fair amount of structuring and transparency and communication with students, maybe even especially the highest achieving students. One of my favorite students of all time went on to become a high school social studies teacher. And she did a fantastic job in the intro class with the unessay and then told me like a year later, “You know, I hated that assignment when I did it.” And it was because she was very, very nervous. She was a straight A student who was really nervous about doing something different. But she said, “Now, it’s my favorite, looking back.” So providing that structure and support and transparency, a lot of careful thought on the front end.

John: And one of the things we’ve been talking about in one of our reading groups, Mind over Monsters this semester, at both of our institutions, is the importance of autonomy, in terms of increasing student motivation and giving them more sense of control. It sounds like it’s really helpful in motivating students.

Jessamyn: Yeah, I think so. In both of our disciplines, criminal justice and history, there is a major learning curve to helping students see themselves as having agency in this particular discipline, especially the way they may have interacted with it before. And for me, coming from a lot of their previous experiences with history has been learning what happened and not actually doing history or thinking about those processes and the unessay could really help them move towards that particular desirable learning outcome.

Rebecca: I think one of the challenges that faculty who have been engaged in unessays before see before they enter in is the complexity of having students doing many different kinds of things at one time and managing those projects and also assessing those projects. Can you talk a little bit about how you help provide that structure for yourselves and for students?

Maggie: Yeah, so for my criminal justice class, I spent a lot of time the summer before I implemented this project, creating very detailed instructions and expectations. Jessamyn, I went to your website, and I use some of the self assessment tools, and I adapted them for my class as well. It was a very mindful process to think about how to give students the structure they need to feel confident in their ability to carry out this project, but also give them enough flexibility and creativity to take this project where they want it to be. I also spend a lot of time reading about different alternative grading practices to also incorporate in this, so I use kind of a combination of ungrading and specifications grading as well to help students assess their work. And I met with students quite often throughout the semester, particularly we had a midterm and a final exam meeting where we talked about the strengths of their projects, where they needed to go and what areas they needed to work on, whether it was engaging more deeply in the research and how to communicate that effectively in their projects. It was a lot of work for the faculty, but it was worth it in my opinion. It wasn’t so onerous and it wasn’t boring. It was exciting to meet with students, even if it meant spending my entire midterm week in the office with the students. It was almost every minute of my day for an entire week doing this, but it was, I think, a unique opportunity for making those connections with my students.

Jessamyn: I have been lucky. I have a relatively light teaching load and small classes. So that did make a difference in how I thought about structuring and implementing the unessay. There would be challenges on a larger scale. I would also say that there’s components of the unessay, that it’s not just do anything you want, and then nothing, and then in the end they turn in something. No, there’s stages and components like usually a proposal process and I also was able to meet with students. You could meet with students in small groups as well, if it’s a larger size class to go over the proposals, and many unessay assignments also do have a written component like a bibliography, an intro. At the midterm and end there could be a reflective assignment or sort of summary of work achieved. I did co-create with students the assessment guidelines for their projects. So taking the autonomy and agency a step further, I didn’t leave it totally up to them, we worked together on it, but asked them, “So this is your idea, this is the project you have in mind. How will it show these important skills? How will it show that you’ve met these course learning objectives?” And we work together on it. And that part is always great for me, I just really enjoy trying to de-center myself a little bit as the instructor which again, is something I can do from a place of certain privileges. I have tenure, I’m white, I’m cisgendered. But trying to enlist students as partners more, decentering myself a little bit, and asking them to reflect on what would show that you have achieved this learning outcome with this unessay assignment. A nice bonus is always students, at least one student, will say this is so hard, teaching must be really, really hard. [LAUGHTER] So bringing them in on that assessment process can help them even reflect on what does grading and assessment mean? What would show me that you’ve successfully achieved this learning, this desirable outcome?

John: Do either of you use rubrics for the unessay assignments?

Jessamyn: Well, when I co-created assessment guidelines with students, some of them used a rubric form. A lot of my students found rubrics helpful for things like deadlines. And I did set deadlines for certain benchmarks. And that was something that was easy to map out and help them structure their work time and stay on track as well.

Maggie: Yeah, I created a checklist with that self assessment that was on Jessamyn’s website, geekypedagogy.com. I did have students go through that and rate themselves, and then we got together in a meeting to discuss where we felt the accuracy of each of those points. But I think it gave them some comfort in saying, “Okay, I know I did multiple revisions of this, and I incorporated all of the feedback I was given. And here’s the changes I’ve made, or here’s the areas that I think I probably could have done better, or I could have focused my time a little bit better in this way.” And it just gave them, I think, a nice guideline to be able to talk about the strengths and areas of improvement for their project.

Jessamyn: I’ve used rubrics before, when I was co-creating the assessments. That’s fine with me, if that’s what they want to use. I will say that this part of it, while being very cognizant, like I said, of my own privileges and positionality and aware of that, your teaching context can really shape what kinds of assessments you can implement, for sure. But I also think that it’s at least worth considering the idea of our academic egos and our investment in our discipline and expertise. And what the unessay may challenge us to do is let go of the thing we know how to do absolutely better than anything in the world. So as a historian, a trained historian, I know how to assess student writing better than any other task I do as a teacher. I know so much about it. And it’s easy for me to do, not joyful, [LAUGHTER] but not difficult. It doesn’t push me to think about anything outside my comfort zone. And when I assign an unessay, I have to let go of that and think about things differently. And again, keeping in mind that I know bandwidth is limited, the incredible pressure on a lot of people in the pandemic pivot and for all various reasons, our limited energy, it might be too big an ask, too big a lift. But I would encourage listeners, if it’s possible, to consider maybe it’s not about me. [LAUGHTER] It’s not about us, and what makes us most comfortable as the person with the PhD in the room. That being a little uncertain. how can I assess did students meet this learning outcome with a quilt, with a board game, with a podcast, that it’s okay for us to feel a little bit discomforted. That’s about maybe getting out of the students way a little bit.

Rebecca: When I hear folks talk about unessays, both of you as well as others that I’ve heard talk about unessays, I see a big emphasis on process and the behaviors of a discipline, rather than necessarily the outcomes of something. So often in a research paper, we might assess on some of the polish of something like that, rather than really seeing always the process that goes into the thinking. But I feel like a lot of times with these projects, you end up knowing a lot more about a student’s thinking, because you’re talking with them about their projects in an ongoing way. Can you talk a little bit about that process piece with unessays?

Jessamyn: Well, one thing that jumped into my mind, Rebecca, when you were asking your question is writing a thesis statement for a research paper. Now, this happens to be something I’m incredibly good at helping students do, crafting their thesis statement. But, just like you described, it’s often so much about the academic writing skill that I’m helping them with, and when they have to write a thesis statement for the research paper they’re working on, that’s a great skill to have, if you’re going to be an academic. If you are engaging in scholarly writing, you have to know how to do that. And for a few students, that’s important. And that’s not like written communication skill. I’m talking about the highly academic scholarly skill of summarizing the argument you’re about to make in a paper. So for students who want to go on to be graduate students, I’m all in. We’re going to sit and we’re gonna go through this word by ever loving word to figure out the best way to state your argument. For a lot of my students, though, is that the best investment of their time, and the best way I can help them? There might be other ways to, just like you said, have them articulate and demonstrate their thinking, which, and I’ll just throw it out here in the age of generative AI and ChatGPT, this is at the heart of how we help students navigate is their original ideas really matter, their ideas, and their thinking matters. So if I can support a student being able to demonstrate their thinking and ideas in a way that matters to them, that they feel articulate doing, then, mission accomplished. That’s what I want. That’s what I want for students.

Maggie: Yeah, and I would say that, in my class, the 20- to 30-page research paper does serve students who are going to go on to graduate school. But like said, I don’t know if that is the best way to help them make the connection between what we’re doing in the classroom to what they are going to be doing on a regular basis in their various careers that they find themselves in. I wanted to make this project less about a means to an end for them, I wanted them to really think about how they start with an idea and how if they spend enough time thinking about that idea, and really focusing on how to communicate that expertise that they gained throughout the research on this idea, that they can create something that they’re proud of and they are hopefully less focused on the grade at the end of the semester and more focused on “Okay, I was actually able to start in one place, develop that project, develop this in a way that I can now go to my future employer or in an interview and say that I’ve developed these skills along the way that are mine and mine alone. They’re not ones that have been imposed on me by an instructor.”

John: We have increasingly diverse student bodies in terms of their prior preparation. Some have come through and had lots of practice writing throughout their high school careers and have developed those skills, others haven’t had as much preparation in writing. And we have lots of courses, though, that give them that but writing skills are not always one of the most important learning objectives for many of our classes. And it seems to me that perhaps using unessays can help provide a more equal playing field for students who may not have as strong of writing skills, but might be fairly comfortable creating videos or might be able to engage in some other creative activity that can demonstrate what they learned much more effectively than if they were constrained to having to do it in a written format.

Maggie: Well, I would say that was my experience for sure. I had a group of students make a series of TikTok videos and so they were quite excited about their ability to use something that they are on almost every day, if not every hour of their day, to use that as a platform to communicate the time topics that they were most interested in. And so it did mean that I had to learn a little bit about TikTok, I had to create a profile so I could follow my students and be able to see the videos that they had created. There’s an up voting system, too, I had to navigate [LAUGHTER] but yeah, in terms of equity, it does give students the ability to make decisions about how to present information in ways that they’re comfortable, but still challenging. I think that no matter what mediums students picked, it was challenging regardless. So it didn’t feel like students were just picking something that “Oh, well, I do this all the time.” It really was a strong motivator. And I think in a lot of the conversations we’ve been having in our reading groups about how to channel anxiety about something into a more direct way to accomplish a challenging task. I think it gave even students who are perhaps underprepared for academic writing, or find a lot of anxiety around that to shift their focus away from what they envision that they struggle with to then start in a place of, “Oh, I know I can do this, and I can communicate it in this way.” So I think it gave them some of that confidence back, that perhaps they find disappears when we impose a pretty lengthy paper or academic medium on them.

Jessamyn: There’s two aspects of the unessay and equity minded teaching I’ve been thinking of. One is the universal design for learning concept of providing some choice. It doesn’t have to be unlimited, it doesn’t have to be infinite choice. But providing some choice does increase accessibility. The second kind of key equity minded principle I have in mind with the unessay is that we should not be grading and assessing people’s life circumstances, we are grading and assessing their learning and their ability to show that they have reached certain benchmarks and outcomes. Just to put it in here as a caveat, I do think all kinds of writing is very, very important. Writing skills are very, very important. And a lot of the unessay assignment as I use it does include some clear written communication, because that is a skill we need in this world of ours. So the equity piece comes in though, that if I’m going to be assigning a grade assessing student writing, then it’s on me as the instructor to teach that and create the conditions in the classroom, the practice and the feedback, and more practice and making mistakes, and more feedback to do that writing piece. And if I’m taking into account the wide variety of academic backgrounds, what needs to happen in the course, for students to be able to demonstrate their learning accurately via a written essay, that’s a lot of writing work that I’m going to be doing to provide that equitable classroom. So the unessay as an option, that’s something a lot of people do offer the unessay as an option alongside a traditional essay option, the unessay as an option could increase equity in that way. I think it does convey the idea that our diversity is an academic asset. That all said, it’s interesting that the unessay could also exacerbate inequities without structure, without support. Students who have had very rich and varied past educational experiences may leap at the opportunity and see it just solely in terms of its benefits. Other students with less of that in their background might be extremely nervous about it as a whole new frontier for them. So without some structure, transparency, without all that prep work, Maggie, that you were describing, it could actually exacerbate some of those inequities.

John: And I think the scaffolding that you both described is really important in relieving some of that uncertainty, because I know some students when I’ve given similar types of assignments are really anxious about it and letting them do it a little step at a time and master that step seems to help relieve the pressure on them by just reducing the uncertainty.

Maggie: I know one of the things I incorporated in the project, and I think Jessamyn did this as well, that I gave students unlimited rewrites on any of the written work. So I did have a bibliography and I did have a proposal and a sort of artist’s statement reflection at the end of it. I think, again, I took that from geerkypedagogy[.com] and I gave students the opportunity to just keep rewriting and rewriting. And yes, it was a bit more work on my end. But it was so refreshing that students were actually taking my feedback into their work and they were making improvements. I think that whenever I’ve done drafts in the past, they might go through and make just the easy corrections and leave some of the harder things aside. But in this project, because we started a lot of those elements very early in the semester, we spend weeks and weeks just revising. And it really did give students, I think, a opportunity to make sure they fully were understanding and achieving those outcomes.

Jessamyn: Yeah, that point Maggie about your feedback actually being read and applied is so important for students and for faculty. It’s discouraging and disheartening to provide students a ton of feedback and then feel like it hasn’t been read. It’s up to us, though, to provide the opportunity for students to apply the feedback. If it can’t be applied, it’s very difficult to take it seriously [LAUGHTER] and to see it as meaningful and relevant. And I think another aspect of what you are describing that listeners might be interested in… if we haven’t totally convinced you yet [LAUGHTER] to use the unessay… is that it does get students’ attention, it cuts through the static. We get into a routine we’re just trying to get through, check that box for students who are overworked, overloaded, for whatever reason. I really feel like it kind of cuts through a fog that many of us, students and faculty, are in just by asking us to do something a little bit different.

Rebecca: Sometimes in those cases, and you guys have alluded to this before, that doing something different if they’ve not done it before, is risky and scary. And so all that scaffolding that you’ve built described as so important for students to be able to take that first risk and maybe take other academic risks in the future.

Jessamyn: Yeah, we can’t underestimate the impact of our traditional grading systems and traditional school systems where the stakes are sky high. You can’t overestimate them, financial aid, everything, is so tied to a student’s GPA, their families support, staying in school is often tied to GPA, their own sense of their selves as students, who they are, is defined by the grade they get. So asking them to, Rebecca, like you said, consider the process, not the end point. It is a big ask. So we can’t expect instant student buy-in, that they will immediately jump on board and with ease and joy, even if we are feeling that way about it.

John: So it sounds like it’s been a lot more fun, even though it may sometimes take more work for both of you. How have students responded? You’ve mentioned a little bit about student responses. But how do students generally respond at the end of the semester? Is it something that they find to be equally joyful?

Maggie: Jessamyn touched on this, but there was a bit of student buy-in that I had to get. And there were a couple of students who still wanted to write a traditional research paper. But in the end, they did a podcast element as well with it. So it was the sort of combination of the two, but it was the first time I had ever implemented an unessay in my classes. So I didn’t have a lot of student examples to show them. But I did spend a lot of time creating my own examples of what an unessay would look like. And I think for each of those scaffolded assignments I created a bibliography and a presentation and I tried to create an unessay myself so that they could see what it would look like. I think that gave them a little more confidence in knowing what was expected of them. But really, it took a lot of conversations with students on why I was doing this, what I thought the importance of shifting away from a traditional research paper, what gains they were going to have from that, and every student, at least to my knowledge, really enjoyed the process and much preferred the unessay over their 20- to 30-page research papers.

Jessamyn: The student resistance you just mentioned, Maggie, reminds me that I mentioned teaching context before, but it is important to underscore that any non-traditional teaching and learning practice and assessment, you can expect greater student resistance or reluctance when you quote unquote don’t look like a professor. So when you don’t fulfill the stereotype of what a professor looks like, if you’re navigating any kinds of intersectional biases, you may have to take into account the student resistance component and student biases. It’s a point that Chavella Pittman has emphasized for like an ungraded practice, that it’s very important to take those considerations into account. I will add that the biggest benefit besides my own grading joy is what I see… it’s extremely hard to measure… I couldn’t put it on my end-of-the-year file assessments of my own teaching, but students gain confidence, students gain some agency, some autonomy. I can see it in how they talk about it. If you can have a showcase or a gallery walk, you can see that they are proud of what they have achieved. When we’re doing senior presentations, that’s always very clear, like students’ sense of accomplishment is very notable. And it’s not that that doesn’t happen with the research papers. Well, it can, but the range of students who express and demonstrate to me that they’ve gained some confidence in themselves as students, as thinkers, as knowledge generators, as producers, it’s really heartening.

Rebecca: So we always wrap up by asking what’s next? Jessamyn, you want to go first?

Jessamyn: I’m headed to the POD Network Conference. I’m gonna do a session, with Chavella Pittman and Thomas Tobin, on faculty development that acknowledges teaching inequities. And I am working on a book on teaching and learning when things go wrong in the college classroom, and I just recently signed a contract for that book with the new series at University of Oklahoma press called Teaching, Engaging, and Thriving in Higher Education.

John: Maggie and I will both be at POD and will try to get to your session. And we’re looking forward to that book.

Maggie: I suppose what’s next for me feels like such a loaded question right now because I’m anticipating applying for a sabbatical. And so I’ve been really ruminating on this question of what is next for me. I’m entering into a mid-career stage, and so I feel like my answer is “I’m not quite sure yet.” But as far as the unessay goes, I do plan on bringing the unessay to my courses next semester. I’m teaching two elective courses on death penalty, and women and crime. And I think really any course, but particularly these courses, would be well suited for an unessay project.

Jessamyn: That reminds me, I am lucky enough, I’m going to be going to the American Historical Association Conference this spring and doing a workshop on the unessay, so any historians listening, I hope I see you there.

John: And we will include links to Teaching History, to that specific volume, as well as to the other resources that have been mentioned here. So check the show notes to find more of those resources. And thank you both for joining us again.

Maggie: Thank you so much.

Jessamyn: Thank you.

Rebecca: It’s always wonderful to talk to both of you

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

307. Career Readiness

Students do not always understand how the work that they do in our classes helps prepare them for their future careers. In this episode, Chilton Reynolds and Ed Beck join us to discuss one institution’s approach to helping students understand and articulate how their course learning activities intersect with career competencies. Chilton is the Director of the Faculty Center for Teaching, Learning, and Scholarship at SUNY Oneonta. Ed is an Open and Online Learning Specialist, also at SUNY Oneonta. Chilton and Ed have both worked on integrating career readiness skills into the curriculum.

Transcript

John: Students do not always understand how the work that they do in our classes helps prepare them for their future careers. In this episode, we discuss one institution’s approach to helping students understand and articulate how their course learning activities intersect with career competencies.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Chilton Reynolds and Ed Beck. Chilton is the Director of the Faculty Center for Teaching, Learning, and Scholarship at SUNY Oneonta. Ed is an Open and Online Learning Specialist, also at SUNY Oneonta. Chilton and Ed have both worked on integrating career readiness skills into the curriculum. Welcome Chilton and Ed.

Chilton: Hey, It’s nice to be here.

Ed: Thanks, John.

Rebecca: Today’s teas are… Chilton, are you drinking any tea today?

Chilton: I am. It’s afternoon here, so I’ve moved to iced tea. I make my own decaffeinated, slightly sweetened, peach iced tea for the afternoon.

Rebecca: Sounds nice and refreshing.

Chilton: Yes.

Rebecca: What about you, Ed?

Ed: I am drinking a Chamomile honey and vanilla tea, in a very fancy special mug.

Rebecca: Oh, that’s a Tea for Teaching mug. I wonder where you got it.

John: And I am drinking an Irish Breakfast tea today.

Rebecca: Also in a tea for teaching mug. I have Lady Grey, I think.

John: We’ve invited you here today to discuss your work at SUNY Oneonta in making explicit connections between course learning objectives and career readiness skills. Can you tell us a little bit about that?

Chilton: Yes, we’d love to. And again, thanks for having us. We’re excited to be here to share this project that we’ve been doing. We just finished up with our first cohort where we are really trying to help our students make really clear connections between what’s going on in the classroom with career competency skills that they’ll be using after they leave college. So the focus of this program was really on helping faculty build into their courses, times to allow students to reflect on what they’re doing in the classroom, and really say explicitly: “Here’s the skill that we are trying to build towards. Here’s what we’re doing in class. Now, as a student, practice actually making that connection. We want you to either write that and think about it in writing or say it out loud, and practice saying out loud, so that those connections can become as strong as possible after you leave this class.” The focus of this was on lower-level classes, we specifically targeted lower-level classes, because we thought that by the time they’re getting to their senior seminars, they’re doing that in the class already. But we don’t have these conversations in our 1000- and 2000-level classes. And so the more we can do this in our lower-level classes, hopefully when they get to those upper-level classes, they can say, “Oh, yes, I do remember talking about technology skills or communication skills early on and I can make connections now between what happened in that class and what’s going on.”

Ed: Yeah, the big thing that we always are talking about in the instructional design field, in the faculty development field, we’re talking about authentic learning all the time. But I joke sometimes when I say like if a student completes a course built on authentic learning, but can’t talk about it in an interview, or articulate it to themselves or others, did it really happen? And this is our practice. This is us saying, “if we’re going to do all the effort to make sure that our courses are built on authentic learning, we’re building authentic tasks into them. Let’s go ahead and do the next step of reflection, of practicing, so that students are prepared to speak about it.”

Rebecca: Can you talk a little bit more about how you rolled it out to faculty, because you’re talking about working on it through the center, and then getting faculty to adopt it in these lower level classes? Can you talk a little bit about those details?

Chilton: Yes. So we had started with a call for faculty. We actually had gotten a grant, there was local money from our institution to be able to do this where our incoming president had created… we didn’t have a strategic plan at the time, so he created a initiative called “regaining momentum” that was very much focused on re-engaging our students, both incoming students and current students. And so one of the focuses was on career readiness. And so that was kind of “how do we help our students make those connections?” So we had applied for a grant, we received the grant, and in doing that had promised that we would do this over three years, the first year being our first cohort. So we actually put out a call for proposals and went to a couple of faculty that we knew were doing some of this from some of our previous work, and said, “Would you be willing to be a part of this?” and then also have the full call for everybody across campus, and we were looking for 10 faculty, and I think we had 11 proposals to begin with. After that we went through the team that was built from across campus. We can talk about that in just a second too. But they had a team that would review those proposals and then said, “Yes, we had a cohort of 10,” which is what we had funding for the first year of the cohort, and then went through the process with them over the year.

Ed: In our center, we’ve really been thinking about how we can focus on the student experience. We’re in a transition phase right now, we used to be known as the Teaching, Learning, and Technology Center, and we are transitioning, and as of July 1st, we’re now the Faculty Center on campus. We were thinking about how we stop leading with technology. We always were thinking about teaching but we wanted to lead with that. And one of the things that we were doing was we were focusing in on the AAC&U high-impact practices. And we went through that long list of high-impact practices and said “okay, what fits into the work that we are already doing as a center?” and kind of identified some of them, so we had already been doing sessions and cohorts of project-based learning with our faculty members, we had already been investigating and helping build ePortfolios. And we always saw ourselves as the collaborative learning people. So what we wanted to do was create a cohort of people that were thinking about this and tie it to a goal that we could keep coming back to, and have these faculty meet with each other throughout a semester to really create a community around a central idea. And that’s where the idea really came from was to keep reconnecting through the semester and focus on building that community versus the sometimes one-off presentations that faculty development can sometimes feel like.

Rebecca: Can you talk a little bit about the career readiness competencies that you’re focusing on?

Chilton: Yeah, when we started this application process, we were connecting with other groups across campus. And one of those that when we talk about career development should be the Career Development Center. So we reached out to them and they talked about how they were using the NACE career readiness competencies. NACE is a national organization that is connecting what’s going on in the classroom to careers afterwards. There are eight competencies that are a part of those and they align a lot of their work specifically with that. Additionally, we found out that some of our co-curricular activities also aligned with the NACE competencies. We have a Lead program, which is a leadership program on our campus, and that uses the NACE competencies as well. The School of Liberal Arts has a program going on right now where they were trying to do a lot of this similar work outside of the classroom with students helping them connect what they had been doing in their classes with what was going on through the NACE competencies. So we found there was a lot of work already happening on campus, and so we really wanted to make sure that we aligned with that as well. What we like about NACE competencies is it really aligns with a lot of the work that goes on in our classrooms. And that’s what resonated with us, and that said, we’re focused on what’s going on in the classroom, how we can help support faculty in doing more useful work inside the classroom. And so we thought about how the NACE competencies really do that. So we think about things like professionalism, communication, critical thinking, teamwork, technology, and leadership. Then there’s equity, inclusion, and career and self development. Those are the soft skills is that word that got used a lot in the past to kind of say, “Yes, we do these things, but we didn’t really help students make those connections between what’s going on.” So we felt like it was a great framework to take into the classroom and say, you’re doing this as faculty, you know, you were doing this, but the students don’t always know that they’re doing this, how can we be able to help do that? The other thing I’ll follow up with that is, as we were exploring this more, we reached out to the POD Network, and actually found out [LAUGHTER] from SUNY, there was already work going on some of this as well. So our Center for Professional Development, has a whole certificate program that’s around connecting career readiness skills into the classroom and our use of the NACE competencies as a part of that as well. So it was really a lot of tie-ins that we saw really strong connections between what was happening on our campus and things that were happening locally.

John: We have talked about that, to some extent in our previous podcast with Jessica Krueger, and we’ll include a link to that, in the show notes.

Chilton: And one thing I follow up with that, John, is we have a couple of pre-professional programs. And this seems to fit really well there, like career readiness makes sense when you have a pre-professional program that’s preparing you for a specific program. We were also trying to reach into our liberal arts programs, into our science programs, into lots of other programs that might not be as focused on a specific profession, but still are connecting into these career readiness competencies.

John: And since we’re doing these things in the classes anyway, it’s nice for students to be able to recognize that these are skills that are going to be helpful for them in their future careers. And when they can see that, I think that may help provide a little more intrinsic motivation to engage in these practices and develop those skills. How have students responded to that?

Chilton: So we are in the first year of this, and this is one of the things we were reflecting on as we were preparing for this in that we realized our first year was focused on what faculty are going to be doing. As Ed said, we’ve been working with some faculty on this, that have been doing this on a smaller scale. But as far as this program, we’re looking forward in year two to really hearing from students and hearing how that’s going to go, so we’ll have to provide some feedback and liner notes later on to let you know when we hear about from the students.

Ed: Yeah, I’m gonna lead a committee to do the IRB and create some surveys to send out to students that are part of the program and have a little bit more of that student voice that we can report back on. Because I think that’s really important. It grew out of a proposal like that, that I’ll talk about a little bit later. But we had done that student interviews and student feedback once before, that really helped create this framework that we were really trying to set up with now a cohort of faculty members.

Rebecca: I really love hearing that you’re using NACE across your institution in different spaces. So you mentioned that Career Development Center is doing it as well as your center. Can you talk a little bit about how that collaboration is working?

Chilton: Yeah, so we really see this as a partnership. And it’s one of the things that we really tried to be intentional early on. Because when you say career readiness, that is a Career Development Center thing, and we don’t want there to be any perception that we’re trying to take over what they’re doing and we want to be able to just support them so that when students come to them, they are more prepared. A part of the original proposal was going to the Career Development Center and saying, we want to do this with you, would you be willing to partner with us? We can do more of this in the classroom. It was very much a partnership, it was very much us wanting to say, “What is it that you do in the Career Development Center, and then also, where can we help support you?” And then make sure that we feed into what you’re already doing. So it’s not any appearance of us coming in and trying to take over your programming, but just help our students be more prepared when they come to your program.

Ed: Actually, we had a great day at a winter workshop where the Career Development Center sat with our faculty and said, “Here are some of the things that we are already doing, here are the services that we’re purchasing, here are the things that we do at one-on-one consultations, here was what it could look like if you invited us into your course.” And some of our faculty members did that and invited the Career Development Center into their course to speak to them. And some of our faculty members were doing other things that incorporated the competencies but didn’t necessarily incorporate an outside group like the Career Development Center. So we had a wide range, even among our cohort of what they were doing. During that winter workshop that I was referencing, we actually brought in an outside trainer. And that was really nice. Chilton mentioned that the SUNY Center for Professional Development, the CPD had already been doing a four course sequence on the NACE competencies, which was really meant for a variety of professionals, it wasn’t just faculty, but the instructor that came highly recommended to us was Jessie Stack Lombardo also from SUNY for the SUNY Geneseo Career Development Center Director. And she came in and did some workshops with us and the faculty thinking about what are the small things that we can do in our class that helped students reflect, that helps students make those connections?

John: Could you tell us a little bit about the impetus for starting this program?

Ed: Yeah, so even before the program, we were, of course, working with wonderful faculty members here at SUNY Oneonta. And one of the things that we’d been doing quite a bit was thinking about making websites and ePortfolios, having opportunities for students to build their own web space, build their own web presence. So even before the cohort happened, we had one great instructor that said, “Hey, I would really love to be thinking about building ePortfolio projects into my course, would you help me do that?” During this time, John, you know, we were doing the SUNYCreate, a domain of one’s own initiative. We were giving websites to students. That was a technology-focused initiative. But we were doing a lot of these things already. I said, “Yeah, let’s come in. Let’s do that.” I was invited into that class several times. And we were so proud of this course, the way it came out. I want to give such a big shout out to Dr. Sarah Portway. She later went out and won the Chancellor’s Award for Excellence in Teaching based on a lot of the work that she was doing. They were building a fashion magazine online, the students were taking the articles they submitted for that fashion magazine, and also bringing them back to their portfolio and showcasing them on their own sites in addition, and she said, “Hey, why don’t we take this thing on the road? Why don’t we go to the AAC&U’s Institute on OER and ePortfolios.” And we said, “Okay, let’s do an IRB, get some student feedback from that, to bring to the conference so that we have that student voice when we go through.” And the feedback was fantastic. Students really responded to it. It was a wonderful presentation. But we were also starting to realize during those interviews, it wasn’t a negative, but it wasn’t all positive. Students were still not making all the connections between the skills that they had done and things they had practiced and the skills they had acquired, and being able to articulate that. I have this memory of a student saying to me, “I wish I could have put this on my portfolio, but it was a group assignment, so I can’t put a group assignment on my personal portfolio.” And I remember just kind of stopping the interview format and saying to her, “Oh, I would absolutely put that group assignment on your portfolio if you’re proud of it. I would absolutely describe what it is you did to contribute to that. group atmosphere and talk about how you can be a successful collaborator and describe how you work in team environments. And then put that thing that you’re proud of, that artifact that you’re proud of, on your portfolio, but also with the framing of what it means for you to be a good teammate and what it means for you to be a good collaborator.” And the student said “Oh, I never thought about it like that, I guess I could do something like that.” And Dr. Portway, being a fantastic instructor, never being satisfied with how things went in the last class was kind of like, “We need to think about this a little bit more. We need to be more explicit. We’re already doing all these authentic assignments. And at some level, it’s hitting. And we definitely want to keep going down this road. But on some level, we are missing something in helping those students make those connections. What do we need to do in the classroom activities, in the way that the assignments are presented, that helps walk them through that to make them just a little bit more prepared, because the authentic skills were already in the course. They just needed help making that connection.” And that was really the thing for me that I walked back from that experience and knocked on Chilton’s door and said, “We need to be doing more of this, and we need to be doing it not one at a time, but with groups of faculty members.” And that was really important to me.

Chilton: And what was interesting to me was that to carry that on a little bit more, when we first had this proposal ePortfolios was in the proposal title. We were really focused on we want to do ePortfolios for everyone. And some of the feedback we received was “Yes, ePortfolios can be a part of that but this could be a much wider conversation,” which is again, how we got back to NACE. There are these bigger frameworks that we can be a part of. So we went from, “Yes, here’s this great tool to no, no, no, let’s look at it from a framework perspective.” And now we’re at the point where we’re like, yes, some of the projects will be ePortfolios, some of them will be other things. And that’s okay. And that was bigger than the tool. This is about helping our students think about what they’re doing and helping them connect to things that will be useful for them after they leave college.

John: One quick follow up, you mentioned that you have groups of faculty who worked on it. As I understand this, this was a faculty learning community that you put together, where faculty received some slight funding or a small stipend as part of the participation. Have you done any work there with entire departments in revising their curriculum yet?

Ed: No, we haven’t done the departmental level work yet. Right now we’re focusing on coalitions of the willing, having faculty who are interested in these types of things. One of the hopes is that after doing three cohorts, having worked with multiple faculty one year, then the second year, then a third year, new faculty each year that we can get to a point where we’re ready to have a bigger discussion. There was one participant in the group that was really focused on making a freshman ePortfolio with the explicit reason to keep contributing to it throughout the program. And I think that shows a lot of promise. But we’ve still got to do some work to get the buy-in from the rest of the department to make sure that it gets used. So I mean, there is a lot to be done there. And it’s one of the things I’m hopeful for the future.

Chilton: To add on to that, one of our goals out of this is to be able to build a repository where we can share and our hope is that we can have enough examples that when we go to a department, we can say, here’s some small changes you can make. Ed had mentioned this earlier, we want to be able to have a breadth of here’s some small tweaks you can make. Or here’s some larger things you can do. And be able to have some examples that are multidisciplinary, that are a wide range of both implementation needs, as well as examples from different departments so that when they go to a department, we can say this doesn’t have to be a large change, it could just be helping make some small changes to help those students make connections.

Rebecca: So I wanted to follow up on an earlier point that you were making, from an experience that I’ve had as an instructor, and I’m sure many other instructors have had is you work really hard to make these kinds of career-ready activities, things like professional email writing, and portfolio projects, and team projects, the list goes on. We do many of these things as instructors, and then you inevitably have this conversation, a one-on-one maybe with a student. And you just realize they have no idea why they were doing any of the things and you’re like, “Oh, I failed the student clearly [LAUGHTER]. I could have done a better job.” And so it seems like frameworks like NACE could be really helpful, both for instructors and students to just be more explicit about those things and to practice talking about them. Can you talk a little bit about that piece of the puzzle?

Ed: Yeah, I think it’s so important to have small opportunities to embed a skill or embed a practice in there. So I’m going to start off with a very small thing that I think anybody could throw into their class. At the end of the course, it’s at reflection time, we want to talk about what you learned. Let’s take a moment and think about a common interview technique is the star interview technique, you’ve probably heard of it, where you describe a situation, then you have the task that you were assigned, the action that you took, and then a result, explain that, say “Hey this is how a lot of times we make sure that we have an action-oriented response to an interview question.” Now talk about this course, using the STAR method. What situations did your instructor put you in? What were you asked to do? What did you do in order to be successful in there? And then is there anything else you want to share about it? Are there next steps that you should continue to do that your instructors put you on the path to? Or are there things that you’ve realized about yourself that you need to continue on with for the future for the next thing? That’s so simple, it’s not rewriting an entire course. Yet, it’s a little opportunity to say, this is important, and what we did had meaning, and take a moment to integrate that into your context. How will you talk about this course in the future.

Chilton: What was interesting to me when we were doing this was when we first started out, we list a whole bunch of sample outcomes that get at what you were talking about, I’m going to do this email, I’m going to have them do this thing and it’s going to be great. And as we got to the conversation with our faculty, we realized that what we were missing was really creating the places for the students to practice making the connection. We have to practice the skills all the time, we’re like, “Yes, we do this.” As the faculty member, we understand that there is a connection between this and career readiness but unless the students are actually practicing making the connection, not just doing the action, but making the connection, then it doesn’t always stick for them. And so that’s where we started to shift from, what do we want the faculty to do? How do we want the students to practice this so that it does stick for them? So it is meaningful for them in a way that they can think about it again, hopefully, a year, two years from now, when they’re finishing their college career and starting to think more about career readiness. That was a shift for us of what is the faculty member going to do to how do we help the students really intentionally practice what they are doing, practice talking about what they have done, and making that connection to, in this case, the NACE framework, because we thought it was such a good framework to talk about.

Ed: I feel like we’re saying NACE too often. So I feel like it’s always helpful to be a little more specific. So let’s talk about communication. We’re teaching communication to students all the time. One of the key aspects is audience. So have the conversation with your students. When we communicate to different audiences, we use different standards. So part of the reason why I’m asking you to write a more formal paper, in research style format, is I want you to be prepared to speak to other experts in your field. But when we shift to the oral presentation, I want you to adjust your language, so that you’re speaking to a non-expert, you’re speaking to your future colleagues, you’re speaking to a potential customer. And when you make that switch, make sure it’s intentional. And then at the end of this course, I’m going to ask you to reflect on that to think about what choices you made when you were speaking to someone who you expect to already understand and be embedded into the discipline and someone who you do not expect. How is it different when you talk to a colleague versus when you talk to your friends and family about what you do. That’s an important communication competency. So let’s talk about it and intentional choices that we can be making.

John: How many faculty members and how many departments were involved in this project so far?

Chilton: So we had our first cohort, and in that cohort, we specifically targeted to having 10 faculty. But we were very specific about trying to have faculty from as many departments and as many schools as possible. So we have three schools on our campus, we ensured that we had representation from all three schools, we ensured that we have representation from multiple departments. So in the end, we had nine different departments as a part of this. We did have overlap in one department, from two of our participants. As we said before, we did focus and said in the call that we wanted you to work with a 1000- or 2000-level class. And so that was part of the call as well. So we actually had a couple people that applied for this that were planning on doing this in a 3000-level class. We reached out to all of them and said “Do you have any lower-level classes that could be part of this?” Two of them said yes and one didn’t, which was one person that we weren’t able to take in. We’re focusing on in years two and three, again, lower-level courses, and going to try to continue to have faculty from as many different departments as possible, so that when we get to the end of this, again, we have a nice repository of examples from as many different disciplines and as many different schools as possible.

Ed: And we can invite some of the cohort I faculty back as mentors, and we can incorporate them into year two in a different way, as we continue to try to build a larger community and push a conversation that we think needs to happen on campus.

Rebecca: Can you talk a little bit about what was expected of a faculty member who was accepted into the program?

Chilton: So we’ve spelled that out upfront, we had already been planning on our campus, what we call the SUNY digital learning conference that was focused on open and public education. And we purposely built in a track in there that was about career readiness. Originally, as we’ve been talking about, we were focused on ePortfolios, and so I thought a lot of them will be doing ePortfolios. But in the expansion of that, we wanted to make sure that we really talked about how we can make connections. So we said that we would pay for the faculty to be a part of that conference. So they attended that conference in November of 2022. And that was the first part, kind of the kickoff for this cohort. We then had a January full day workshop, as Ed had talked about earlier, brought in Jessie Stack Lombardo from SUNY Geneseo to be our speaker for that, and she wasn’t even a speaker she really planned the day and it was very highly interactive with those ten faculty. So as Ed said, we have a staff member from the Career Development Center was a part of that and presented locally, Jesse then talked about some different frameworks to be able to do including NACE and how you can start to think about both small changes that we made and large changes. And then we had said in the call that the expectation would be that by the end of the Spring 23 semester, they would turn in a revised syllabus and examples of work that they are doing to the group. We realized that wasn’t specific enough. So we then created a rubric that focused in on three specific areas of what they would need to do. First part of that rubric is what would be the changes they would make in their syllabus to really spell out what are the NACE competencies? Are you focusing on all of them? Are you focusing on one of them? But didn’t have to be a lot in there, but we did want to have it be addressed in their syllabus in some way. And then what is the activity they’re going to be doing where they actually have students practicing and how will the students receive feedback on that? Well, there’s three levels, it’s not present as a part of the rubric. And then we had two levels of “yes, it meets expectations.” And we were thinking again, what are those small changes that could happen, but then also, we had a what’s above expectation, where what’s something if you were really dreaming about what it could be, where could you take it and what could it be, so we kind of want to have “Yes, you meet expectations” that would help us get small changes that would be usable by everybody. And then what could this be look like if you really wanted to really [LAUGHTER] dive into the deep end with it and explore what could happen with it a little bit more, and make it so that it was better for students, not just in the class, but beyond the classroom.

Ed: Yeah, the only thing I’ll add to that Chilton is we also met once a month during the spring semester. So we had recurring meetings throughout the Spring semester. With those rubrics between the present and highly effective, and we’ll share the rubric so that you can put it in the notes if you’d like. We were starting to think about not only did you incorporate the NACE competency in your course, but you were also presented your prompts or your things in a way that gave the students the opportunity to think about future activities they could take, future things they would want to do. And that was really important to us as we were doing it is to not only just create a moment of reflection for the students at that moment, but also to make that connection of, okay now that I’ve had that moment of reflection, now what? Should I be picking out some different courses? Should I be finding an internship? Should I be doing something now to set myself up for success? And so we don’t get that panicked feeling when the student is at senior year and they go into the career readiness and or the Career Development Center and say, “Okay, what do I do now?”

John: What type of incentives were offered to faculty to participate in the program?

Chilton: So we spell that out as pay for their participation in the conference in November. So they were able to go for free and participate. It was on our campus that made it easy for them to be able to go. It was just their conference registration that we covered as a part of this. In addition, we paid them a stipend for attending the January workshop. So officially it was $90 was the stipend to attend the workshop. And then when they completed and turned in their final version of their revised syllabus and examples of activities, there was another $510 stipend. So in total, it was a $600 stipend. But as a part of that final revision, we actually did review their submissions, looked at the rubric and did give them feedback… a couple of people, we said, “Hey you’re missing…” and asked them to go back and do some additional work. So we did hold them accountable to that rubric before getting the final stipend. And so it was a useful and interesting conversation when the leadership team did meet to kind of look at those to be able to say, “What do we like about this? What are we thinking for cohorts two, and three? What might be asked for more specifically next time to make this even more meaningful for our students?” So we’re already starting to think about cohort two and looking forward to that for next year.

Rebecca: Can you talk a little bit about how faculty responded to their participation,

Ed: We take the faculty’s response and the feedback they gave us really seriously. We gave them the opportunity. They had the reflections that they were doing that, of course, we knew who was speaking. But we also gave them some opportunities to give us some anonymous feedback, so that they could tell us how they really felt about us. And we were just really pleased with it in year one. We do recognize that we have to keep honing our message, we have to keep defining what we mean by career readiness, and what we mean by incorporating it into class. We need to have our elevator pitch a little bit more refined and down. Because what’s evolved through this conversation is, we’ve really talked about the skills are already there, but we can be more intentional about it. And we can be intentional in the ways we ask students to reflect and practice in ways that we really believe can be beneficial for students. But that can still be a difficult conversation. When people see career readiness in 1000 and 2000 level classes, some people are bristled or turned off by that because they’re thinking, “Oh, just one more thing that I have to do.” Now we didn’t get that from our participants in the cohort that much, because they applied and they came here on purpose, that was nice to have a group that was really wanting to be here and was willing to try some things with us in this space that we were creating. But overall, I would say that the feedback had been very positive.

Chilton: Looking through the feedback from faculty, I just pulled out there was one quote that stuck out to me that i’ll read quickly that came from one of our professors in our communication arts department, where this professor said, “Students said that they felt more confident.” So this is actually one of the professors we recruited into this program that had been doing this already. This professor did have some experience with students doing something but said that “Students felt more confident in the skills as a professional, and were able to articulate how the experiences they had in my class connected to the expectations and employers would have of them. They also appreciated being told of why we had to do certain projects and to help them transition from college to life after college.” And so I think that really speaks to how the professors enjoyed having time to be able to do that.

John: We only have courses from 100 to 500 levels. It seems there has been a bit of a course number inflation there at [LAUGHTER] Oneonta. That was just a joke. I’m sorry.

Chilton: We were told that everybody in SUNY was moving to four course digit numbers and so we over the past two years, it was like this really big project that we did to move from three digit to four digit numbers, because at least we heard, everybody in SUNY is doing this. So we have to do it. It’s very intriguing that not everybody had to do that. [LAUGHTER].

Ed: And simultaneously, as we were going from three digits to four digits, we didn’t have 400 level classes previously. And the feedback we were getting was that that was seen as a deficiency by some people who are reviewing our students’ transcripts, even though calling all of our upper division 300 level, and that people applying to professional schools would get that explanation they would understand why there weren’t 400 level. Other people who are maybe not as skilled at reading a transcript are like, “Well, did this student avoid all 400 level work?” And so simultaneously as we were adding another digit, we were also transitioning to having 1000, 2000, 3000, and 4000 level classes. So that was a big change that took a lot of curriculum writing and mapping through over the last two years. It wasn’t just as easy as adding a zero on to every course.

Rebecca: Sounds like such a fun project. Sign me up. [LAUGHTER] So we always wrap up by asking what’s next?

Chilton: So we are excited about Cohort Two. We are going to be starting our recruitment in the fall. We actually have a fall faculty Institute on campus. This year is very much focused on what the communities of practice that are already happening on campus and how you can get involved. And so that’s going to be our first, not our first, but that’s gonna be one of our big recruitment pitches for Cohort Two. In Cohort Two, we are looking to be able to include more faculty from a wider range, we are going to be starting to get into faculty that might not have as much experience in doing this. So we are thinking about how we hone our pitch and how we focus this to a wider audience to be able to say “No, this is not big changes in your classes. This is just asking one additional question or allowing space for one additional time for students to be able to practice connecting what you’re already doing to these career readiness competencies.”

Ed: And I would say what’s next for me is this experience has really solidified the idea for me that we need to continue in faculty development centers, to make spaces where faculty can repeatedly come back and interact on the same topic, getting away from that kind of one and done workshop, and identifying major things that we want to return to through the year inviting people into that space to share. Because when those faculty, when they get an opportunity to think and show off what they’re doing, it really is a wonderful spread of ideas. And you get a lot from all the energy in the room.

Rebecca: Well, thank you so much for joining us and sharing how this project’s unfolded at your institution.

John: And you’re both doing some really great work at SUNY Oneonta and it’s great to keep in touch and thank you for joining us.

Chilton: And thank you. It’s a pleasure to be here with you. So thanks for taking the time to be with us today.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

Ganesh: Editing assistance by Ganesh.

[MUSIC]

304. ChatGPT Inspired Course Redesign

AI tools such as ChatGPT have the potential to significantly disrupt how we work and how we learn. In this episode, Don Donelson joins us to discuss a course redesign strategy that could help prepare students for a world in which AI tools will be ubiquitous. Don is a senior lecturer in the Miami Herbert Business School at the University of Miami. He is a recipient of the Spring 2016 University of Miami Excellence in Teaching Award and the Dean’s Excellence in Teaching Award from the Miami Herbert Business School.

Show Notes

Transcript/h3>

John: AI tools such as ChatGPT have the potential to significantly disrupt how we work and how we learn. In this episode, we discuss a course redesign strategy that could help prepare students for a world in which AI tools will be ubiquitous.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Don Donelson. Don is a senior lecturer in the Miami Herbert Business School at the University of Miami. He is a recipient of the Spring 2016 University of Miami Excellence in Teaching Award and the Dean’s Excellence in Teaching Award from the Miami Herbert Business School. Welcome back, Don.

Don: Glad to be here.

John: Today’s teas are: …Don, are you drinking tea?

Don: I am. It’s the same tea that you’re drinking, black currant, and it’s great.

Rebecca: It’s a John favorite for sure. I have a Tazo Awake tea today.

John: Does that mean you’re woke? That may be an issue down in Florida.

Don: …not in a private school.

Rebecca: It means that I couldn’t make a pot of tea. I didn’t have time. So I had to use a single tea bag. [LAUGHTER] That’s what it means.

John: And I am still using the mug from Australia that Clare McNally gave me with kangaroos all over it.

Rebecca: I like that mug.

John: I do too.

Rebecca: I look forward to seeing it in person.

John: Soon.

Rebecca: Yeah, you’ll be back soon, right? A couple weeks.. I’ve got my grad studies mug. We’ve invited you here today to discuss your plans to revise the graduate and undergraduate core courses in critical thinking and business communication at Miami Herbert. Can you tell us a little bit about this course?

Don: So this course started at the grad level, MBAs in particular, in 2008. That’s what I was hired to teach. And it grew with the program, expanding into specialized master’s programs. And then it went out into the undergrad program. And it’s a core course required for all full-time business students, undergrad and graduate. This past year, we had 46 sections of undergrad courses and 21 sections of graduate courses, about 900 students or so in total.

Rebecca: So a really small situation going on here.

Don: Oh, yeah, very small, [LAUGHTER] no problems with scaling or anything like that.

John: What was the typical focus of this course in the past before this revision that you’re working on?

Don: So the course was called “Critical Thinking and Effective Written and Oral Communication,” and it lived up to its name. It was about those three things. At the time that we started in 2008, we called them soft skills. We don’t use that phrase anymore. We like to call them fundamentals, something of the sort. We think that soft skills sends a bad message. But it’s been overhauled three times, this will be the third overhaul since. And the things that we would do in the courses, from the very beginning, the main evaluations would be based on writing memos and giving presentations

Rebecca: Which should be about the kind of communication you’d be doing in business. [LAUGHTER]

Don: Yeah, it’d be based on hypothetical cases, some non-hypothetical cases, the standard Harvard Business publishing 10, 20 page case on “How did Netflix beat Blockbuster?” or something of the sort.

Rebecca: What prompted this big overhaul?

Don: Well, the accreditation body, AACSB, required program evaluation. And it’s sometimes an annoying task that people do just to go through the motions. But what we found, the first time I went through it, was that we actually learned a lot from going through those motions. And so in my department, at least, we institutionalize curriculum audits on a semester basis. And so, in between the fall and spring semesters, we have a shorter meeting, where we kind of look at what happened in all the instructors, faculty teaching in that space in the fall, what happened and what worked, what didn’t work, and we might make some minor revisions. And then at the end of the spring semesters, we’d have a little bit of a longer meeting. And the last few of those had turned out some opportunities for change.

Rebecca: Can you talk a little bit about what some of those opportunities were?

Don: Yes. So we found that this is a challenging course to get buy-in from the students. And so we still haven’t figured out 10% of what we can figure out about teaching in this space, I’m sure. But one of the things that we’ve seen is that it is somewhat of an innovative curriculum, and one of the challenges with being innovative is students haven’t had material like that before. It’s a core course. And so students have to take it. And there’s always challenges with that. But this is a bit of a different challenge. And so I was talking with John the other day, he has some core required economics courses for business students, and some of the challenges that come with that. But this is a bit different in that those students know that economics is a field of study. And they know that people take economics courses, and there’s a textbook and critical thinking and communication. They’ve been taught kind of as separate. They’ve been add ons in other courses, not as a discrete course itself. And so we think there’s some challenges with that. And really, the challenge that we’ve seen, in addition to that, is that from the faculty view, critical thinking and communication are not separate things, they are one thing. And so critical thinking, I would call it problem solving, is really what we’re teaching. But communication is a component of that. And from the student view, we’ve had a hard time getting them to see those as integrated. And so when they do a memo, that’s an evaluation metric, they see it as: “Well, that’s just looking at the writing and not critical thinking.”

Rebecca: That’s interesting. Some of the things that we’ve done in our design courses around critical thinking and writing across the curriculum, my department, which is art and design, is doing some of those same things. We would do projects and embedded in those projects would be things like memos and other ways of communicating as a way to critically think about the decisions that our designers were making on things. But we would run up sometimes against the same kinds of challenges, like how do you really make that feel practical, that’s relevant, and then also keep it interesting. And it helped, I think, in those cases, because it was tied to a project. So is that a challenge that you face in this particular class is because there are these kind of standalone case studies, and it’s hard for students to buy in or get them into a business space?

Don: That’s one of the things actually I think that’s going to be changed is more of an arc to the course. And one of the things that I’m looking at is more integration of assignments. And so things building more towards the other assignments, and so we have skills building on top of each other. But, ideally, the assignments that they’re doing all build towards one culmination assignment, capstone type project.

Rebecca: Where does this course fit into their other required courses? Is it something that happens in the beginning? or in the middle? towards the end?

Don: So that’s partially an administrative question that is dependent on staffing. We see some students wait until the very last semester to take it, particularly the students who don’t have English as a first language, but they can start taking it as early as their sophomore year. But usually, it’s junior.

John: What’s the difference between the undergraduate and the graduate versions of the course?

Don: So the graduate versions are taken on a quarterly basis, and the undergraduates on a semester basis and so there’s more contact time in the undergraduate version of the course. They use different materials, and they’re more in depth. And so much like you would see with undergraduate economics classes, the graduate version of the economics classes might have similar titles, but go far more in depth into the material.

John: So one of the main issues is that students don’t see the critical thinking aspect of it as being important in their writing. How are you going to change your course to focus a bit more on the development of those critical thinking skills?

Don: Well, this is where I need to go back and add more to what Rebecca asked before about what prompted this because of course, ChatGPT prompted a lot of the revisions as well. And so ChatGPT, AI in general, while it’s kind of an independent axis of revision, we were thinking about some of these other problems well, before ChatGPT even became a thing that people were aware of. But they go hand in hand, really. A lot of the problem that I’ve seen with the writing assignments, and why students don’t necessarily view them as critical thinking and focus on the writing, is because there’s writing for aesthetic, and then there’s writing for substance. And if you’re teaching anything about writing, you kind of have to be teaching both. But when you’re teaching both together, the students tend to focus more on the aesthetic. And they connect it back to English composition classes that they might have taken in ninth grade or 10th grade. And those classes are certainly very important. But they’re a bit different than what we’re doing in these classes. And so I think it primes them to approach the course in a way that is not really conducive to getting what we want out of it. And so with AI, well, it remains to be seen, but it looks to me like you don’t need to be teaching the aesthetics of writing so much anymore in a class like this. And so I’m going to experiment with just not. [LAUGHTER]

Rebecca: It’s interesting, because in design classes where we were doing some similar kinds of things, aesthetics obviously always come about, because if we’re doing visual design, aesthetics are a part of that conversation. But we would have the same thing. It was like, “Well, that looks nice. That reads nice. It just doesn’t say anything.” [LAUGHTER]

Don: Right. Yeah. [LAUGHTER]

Rebecca: So it’s interesting that we bump up against these same kinds of challenges across a wide variety of disciplines. And that ChatGPT does offer some opportunity to focus on some different things.

Don: Absolutely.

Rebecca: I’m curious what exactly you’re going to focus on and how you’re gonna leverage ChatGPT in the context of this class.

Don: So I think ChatGPT is an insane, wild, amazing tool. And it’s going to only be more wild, more insane, more amazing next month, [LAUGHTER] or six months from now. But I see it really changing the way that I teach, the way I prepare for teaching. So kind of on the end of creating lesson plans and what it does for me as a teacher and the kinds of things that I can do in a class that I wouldn’t be able to do in a class before without a lot more hours in the day. And then also, from the student side, changing the way that they do assignments. They’re not going to be writing memos outside the classroom in the way that they have in business for decades and centuries. They’re going to be using ChatGPT. And so if, in this course, which is meant to be a practical course, we can’t make it practical if we’re not allowing them to use the tools that they’re not only going to be encouraged to use, they’re going to probably be required. And so if they don’t use ChatGPT in the future, they’re going to have bosses saying, “Why are you spending X amount of hours on this client memo instead of doing something else.” And so we really need to prepare them for that world. There’s been some early research and I’ll get John the citations, but we looked at research over the break between fall and spring, this past year, some preliminary research about the kinds of jobs in the way the labor market is going to be affected in the future by AI and ChatGPT and the jobs that were predicted to be the hardest hit in terms of reduced wages, and just reduced demand are jobs that involve writing and the jobs that were predicted to be the most insulated from Ai were jobs that involve problem solving and critical thinking. And so really, when you look at that research, it doesn’t even give us a choice. Even if we weren’t thinking about making some kind of revisions before, we’d probably need to just on that alone.

John: So is the focus now shifting more to the critical thinking skills and a little bit less on the basic structure of writing?

Don: Yes. And that’s really where even though the impetus for the revisions were independent, in practice, they’re not going to be that independent. And so it really dovetails nicely. And so I’ll give you an example, if a student is writing a memo, where a business is making a decision between two or three different courses of action, and one of the main criteria is the profitability of those courses of action, the structure is kind of guided by the math of profitability. So if you’re not talking about revenues, independently of talking about costs, you’re not proving profitability. And so when we talk about structure in this course, that’s really what we mean. But students very often, because of some of the things I’ve talked about previously, they’re looking at it as far as like the five-paragraph structure. And that’s not really what we mean. And so by being able to focus less on the aesthetics of writing, and more on the substance, I think we’ll be able to undo some of that priming,

Rebecca: …almost like this shift to articulating the decision making…

Don: Yes.

Rebecca: …rather than talking about writing, because articulating, it could be verbal, it could be in written language, it could be in a lot of different formats. But the point is that you thought critically about the issue, and how you made the decision. [LAUGHTER]

Don: Yes, exactly. In presentations, I’ve never had as much of the same problems as we have in the memos. Part of that, I think, it’s because of contemporaneous feedback. My students early on learned that this comment is kind of a trolling comment. And it’s not really meant as a attaboy or attagirl. But, sometimes a student will give a speech, and when they’re done, I’ll say, “I’m very impressed with your public speaking skills.” And they think, at first in the early parts of the class, that that’s a compliment. But they realize that that’s actually not a compliment. What I really mean is “No one would be buying what you’re selling, no one would be buying this stock, no one would be making a decision based on this, but you have very impressive charisma and confidence.” And that’s not really what we’re about, maybe in politics, but that’s a different question.

Rebecca: I’m curious about integrating ChatGPT as part of the process. Are you thinking about requiring students to reveal and discuss how and why they use ChatGPT in particular instances, and how they leverage the tool.

Don: So I think part of it is going to be first showing them how ChatGPT is not a critical thinking tool. And so I think it’ll be kind of walking on the escalator backwards for a bit just so that we can walk forward. It’s not going to be ChatGPT’s here, so you should use it. Go. ChatGPT is like a personal assistant, who is extremely capable and competent, but will do precisely what you tell it to do, and nothing else. The input you give it determines the quality of the output. And so if you go to ChatGPT, and you say, “I’m writing a letter of recommendation for Rebecca, and she was a great student, and she’s applying to law school period,” it’s going to give you about what you would imagine…it’s going to make up some stuff about Rebecca, it might even not get what program you’re in right. It’s not going to use the last name because I didn’t give it one. And it’s going to give you a very fluffy, perhaps disingenuous response. Now if I give ChatGPT a really robust stream of consciousness almost about Rebecca Mushtare was a student in the spring of whatever and she got this grade and she did phenomenal in these areas. And this assignment she really stood out most because of this, this, and this, it might give me a much more usable response that I can then play with. And so I think that’s going to be the first to instruct students on: what it does not do, which is critical thinking. And from there, I think they’ll have to use it however they feel comfortable. We’re still going to have some writing assignments that are scored. But what I’m hoping for is that these changes will make it so that they’re focused much more on the critical thinking parts of it. And so for some students that might look like writing a fairly complete draft on their own, and then putting it into ChatGPT and telling it to edit this for brevity and clean up grammar mistakes or do something of the sort. For some students, it might be much more of a back and forth kind of a conversation with ChatGPT, which I think a lot of students will be surprised to learn that it functions in that way. And when I find myself using it, it’s mostly as a conversation. Like, I didn’t like what you did here, cut that part out and do this again.

Rebecca: It’s funny that we don’t always think about it as a chat tool, despite the fact that chat is in its name.

Don: Yeah, exactly.

John: Before making this major change in your curriculum, have you experimented with any changes in this course recently to put more focus on critical thinking skills before introducing ChatGPT?

Don: Yes, so in some of the sections, especially at the graduate level, since we have so many different master’s programs, when I first started, it was MBA, and pretty much that’s it. Now with where the business world is going, there’s a lot more demand for specialized skill sets. And so we have, in addition to MBAs, we have a Masters of Science in Finance, a Masters of Science in Sustainable Business, so on and so forth. And each of those sections afford some opportunity to take things in a different direction, really, not even just an opportunity, but we kind of want to, to be more responsive to those fields. And so in the graduate sections, we’ve had some isolated ability to experiment with more problem-based learning, which I think ChatGPT goes really, really well with on the faculty end as far as creating problem-based learning curriculum. But we haven’t experimented with the AI component of it yet, really, because it’s so new, and it doesn’t feel like it right now…it kind of feels like it’s 20 years old, but yet haven’t used it. But it’s very new. And so I don’t know about every other institution, but we don’t move at the pace of jets when it comes to curriculum revision at the University of Miami. We move, I think, faster than probably most but still, it takes time. And so we haven’t had the opportunity to do anything with the AI yet. But we’ve revised in the past couple years to focus more on some of the problem solving in some of the graduate sections.

Rebecca: The faculty member in me heard I can use ChatGPT to help me with problem-based learning classes, and I want to know more about that.

Don: Oh, yeah. [LAUGHTER] So if you type into ChatGPT, you have to give it really, really good direction: the who, what, when, where, why…that you are a professor teaching a negotiation class. And it is a upper-level, undergraduate course. And you are going to create simulation practice for negotiation in which you play one role and the student plays the other, and you will create a scenario and interact with the student, but wait for the student’s response after each of your responses. And then at the conclusion, give the student feedback based on what you know about the science of negotiation from a management sciences perspective, as well as a legal perspective. And then you hit go, you will be blown away with what ChatGPT starts to create. And so it will give you a little blurb. A couple of weeks ago, I did something of the sort, and it said, Sally is the owner of a handmade furniture manufacturing company in North Carolina and has been contacted by so and so that owns a furniture retail store. And so and so has been impressed with Sally’s furniture and wants to arrange a distribution agreement. The meeting begins over the phone and so and so ask Sally what her goals are in this arrangement. And then that’s where I would type in and I said my goals are to reach this level of profitability and to have a productive long-term relationship with the other party, and it responded back. So it can create an entire dialogue that you can then ask afterwards, once you tweak it and say, “Well, I liked this part of it. I didn’t like this part of it, write the Python code for this, and it will write the whole Python code and allow you to turn it into a web-based interactive program. It’s really quite wild.

John: So basically, it gives every faculty member the ability to create interactive simulations for their classes, which could be done for pretty much any topic I would think.

Don: Absolutely. In the past that kind of thing was in some courses, probably a bit aspirational. It’s the kind of thing that would probably require some kind of course leave to develop it. And for faculty who become really comfortable with it, it will get to a point where it’s doable within a day or two of a lesson. And so you can on kind of miniature scale, you can do these on a daily basis, really high quality ones.

Rebecca: It sounds like something that we can use in a lot of contexts in higher ed, including if we want to do simulations for interviews for new positions or other things as well, if you’re trying to better understand how someone might approach a problem.

Don: Absolutely. I think that’s a very good application, in fact.

John: One of the things though, that I think has generated some panic for a lot of faculty is the effect that this may have on how we assess student learning. So how can faculty address issues related to the ethical use of artificial intelligence?

Don: Well, I’ve never known any faculty to ever panic over a technological innovation…sarcasm ended. So I think faculty have to assess this on their own, but also part of the community. One thing that I think’s going to be an early problem are faculty doing things in a different way, I think that’s probably unavoidable. And so I say all that as kind of a disclaimer that my approach and what I think our approach is going to be in my department, and even if the disclaimer applies to that, I don’t even know that for sure, is perhaps going to be different than others’ approach. And so since this course is supposed to be so much of a practical course, and the writing is on the wall, no pun intended, well, it’s in the AI software, I view that we really have no choice. And so there’s been a lot of commentary in The Atlantic magazine, a lot of commentary in the higher education journals. And most of that I have seen focused on this question, but using as an assumption that it’s wrong to use ChatGPT. And so the easiest way to make it not a question of cheating is to allow it to be used, and then it’s not cheating. And so that’s the direction that I’m leaning in. And I think, ultimately, for the practical tools, for the practical courses, that’s going to be the direction it goes. But again, I can’t even speak for my own department on that, because we’re so new in this.

John: And that will be an issue, I think, everywhere, as it has been in the past with things like calculators, or smartphones, or even Apple watches, I remember getting all these memos coming in from various places at one point to make sure your students are not using a smartwatch while they were taking an exam, because somehow the answers are going to miraculously appear on that tiny little screen for the test that you’re giving them.

Don: Right. And I think you can’t really separate the assessment design and the student response to the assessment in this. There are going to be some courses, I can imagine, in different disciplines, that they’re focused on more fundamental foundational skills, that it’s going to be more of a challenge for them. Well, I’m not saying that students don’t need to know and learn about the aesthetics of writing; that has to keep happening, but not in this course. And so I don’t know how the faculty in those spaces and really the 9th and 10th grade composition teachers that I talked about before, I don’t know how they deal with it, probably in person assessments, that sort of thing. But for this practical application course, I would view it really as kind of training track runners to hop on one foot. And so that wouldn’t be very practical. And so if you have a cheating or plagiarism or honor code policy that requires them to only use one foot, then it would be plagiarism for them to run on both feet. But that wouldn’t be very helpful. And so I really viewed ChatGPT as the same thing in a practical sense, if you’re plagiarism or honor, code policy defines ChatGPT as out of bounds, you’re training them to run on one foot.

Rebecca: So we’ve talked a lot about the writing component, and really building in stronger structures to focus on critical thinking. One of the other issues that you identified was that students don’t necessarily see the intrinsic value of the course or like get the buy-in. Can you talk a little bit about the ways that you’re redesigning to help with that piece of it as well?

Don: Yeah, and so certainly a lot of students do get that. But it really depends on how intrinsically motivated they are. And I think it requires faculty to kind of sell it somewhat. And so in my courses, I’ve found success with selling it in that way, which I really don’t like to do. And it’s something that a lot of faculty probably think is kind of an icky thing to do. But for instance, I will repeatedly tell students, I’m not here to make myself feel good. I’m not here to make myself feel smart by putting you all down. I’m here to help you all get jobs and to get promotions at those jobs and do well in your careers. And so I will focus a lot on kind of pointing things out as criticism that I also tell them these are not affecting your grade, however, X, Y and Z. And so little things like in a presentation if they go…you know, we have to have time limits for presentations because it’s basic math, we have X number of students, 75 minutes in a class session, we have to have time. And so when it comes to a student has five minutes and they go over, what do you do? I’m not going to take off on grades for that. But I am going to point out for a student that, in some settings, if you’re given a time limit, that’s because the CEO has another meeting five minutes after you start and you will be cut off, not because they don’t like what you’re saying not because you haven’t followed the directions, but because they’ve got somewhere else they need to go. And so a lot of the problem I think, is just students are so focused on grades, to the shock of everybody, [LAUGHTER] that when the things that you’re grading and are affecting their grades are these kind of…and aesthetic isn’t the right word, but they would view grading something like that as a bit ticky tack. And when you’re scoring things like that, it’s much harder to have a serious conversation about the nitty gritty substance, and how, if you’re trying to prove that this course of action is more profitable than the others, and you didn’t provide any support for the change in costs, you really can’t have accomplished your goal. You don’t get the same attention from the students in the same response, if you’re also talking about things like, well, “you went five and a half minutes when you only had five minutes,” or “You didn’t use 10 point font when you were told to use 10 point font and that ChatGPT, with that second example with that 10 point font, if the instructions said 10 point font and the students input the instructions it produce it in the appropriate formatting.

John: And I know in the past, when I’ve graded student papers, I, as many other people do, spend far too much time correcting grammatical errors, reminding them that there’s a difference between singular and plural or the difference between all the various homonyms out there. Might be easier for us to evaluate student work when we can actually focus on the arguments they’re making, and their ability to engage in critical thinking, rather than getting ourselves so tied up in all this minutia, which I always try to avoid doing, but when I see so many errors in student work, it’s hard not to at least correct some of it so that they could become more proficient. In the future, they may not need to have that type of correction.

Don: Yeah, John, I think you really hit the nail on the head there. You really feel like you have an obligation to correct those. And when communication or writing is one of the titular topics of the course, even more so. But I have always felt that you get diminishing returns on the things that you focus on. And every time you’re talking about grammar, you are not talking about the critical thinking, and the grammar does matter. I can tell you that I have lots of conversations with CEOs with HR directors, etc, in the ongoing effort to make sure that my curriculum is responsive to what’s happening in the market. And one thing that I consistently hear is grammatical errors, spelling errors on slides or in cover letters are catastrophic. And it’s not because they’re nitpicky, it’s because the markets are so competitive, that they get a window that’s maybe 5% or less of what someone’s actual quality as a candidate may be. And that’s just something that there’s going to be some other candidate that is just as qualified and equal in every other way that didn’t make grammatical mistakes in their PowerPoints, and so on and so forth. And so it is important, but it doesn’t matter how good your grammar is, how compelling your vocabulary is, if you are missing some of the logical components of the argument, you cannot be correct.

John: And when students get feedback, where they see dozens of comments on it, the easiest strategy is to focus on correcting those small grammatical errors that are riddled through it. You might also have told the student that they don’t have a very substantive argument. But if they’re going to make a lot of corrections, it’s easier for them to focus on correcting the grammar and ignoring the more fundamental problem.

Don: Right. The very first writing assignment I ever did as a graduate student was a 50-page memo and I got back no comments anywhere except for on the front page: “Do you talk like this?” I think probably somewhere [LAUGHTER] in between those two is ideal. But you’re exactly right, that the more that we focus on things like grammar and tense and such things, the less we can focus on the meat and the critical thinking.

Rebecca: It’s funny how that often is the level of polish would be something that goes from someone that’s got like a really high grade to like an excellent grade.

Don: Right, exactly.

Rebecca: Something that’s foundational, that’s often not how our feedback structures work. And even if we keep form and function feedback separate and even weight them very differently. It’s really easy to address the form issues, because it’s almost like a series of checkboxes and it doesn’t require a lot of thought because the critical thinking part’s the hard part. And so it’s funny that even if they’re weighted differently, and to keep the comments separately, students will always flock towards the thing that’s kind of easy to fix. I mean, who wouldn’t? Then it becomes a checklist.

Don: And that’s exactly really kind of where this boils down to me, it’s not to say that those things aren’t important. They’re still very important. But in the world in which ChatGPT is a real thing, which it now is, and will continue to be and only be more powerful than it is, the juice that we get out of spending time in class or in feedback, in office hours, whatever it may be, talking about those sorts of things, is getting much less of a return than it did before ChatGPT. I am not a walking detector of 100% perfect polish by any means. But it seems to me that the product that ChatGPT can produce, in terms of those things that you were speaking about, Rebecca, is pretty dang good and hard to distinguish for me from highly polished products. But again, where it is easy to distinguish is this is a load of crap [LAUGHTER] that is fluff and has no substance to it, but a very polished load of crap, but nonetheless…

Rebecca: It’s pretty crap. [LAUGHTER]

Don: Exactly. It’s very pretty crappy with a nice bow,

John: …which reminds me of some work that I graded just the other night, where spelling and grammatical issues have mostly disappeared in student essay responses since the advent of ChatGPT, but the substance is not always there. And there were many responses that I provided feedback on which said, “this is a really nice response, but not to the question that you were asked to address.”

Rebecca: Yeah, or you spent two paragraphs and you haven’t actually said anything yet.

John: So teaching students how to use ChatGPT or other AI tools more effectively might allow them to be more productive in their learning as well as beyond their college experience.

Don: And might allow us to make for more productive learning environments as well.

Rebecca: So we’ve talked a lot about course content, and what to maybe focus on and not focus on. One of the most important things a course has is its syllabus or course outline. Can you talk a little bit about course policies and the way that you might make change in that realm?

Don: Yeah, so I think you’re gonna have to be more detailed than you probably are used to being in terms of putting language and syllabi, very specific and upfront. And so some of the policies that I’ve seen that I’ve liked elements of and are going to end up including in the syllabi the explicit weaknesses of ChatGPT. It is not a critical thinking device, it will produce responses only as deep or as shallow as you instruct it to. You are still responsible for the critical thinking, essentially, and very explicit in terms of what’s allowed, what’s not allowed. And I think also, it would probably be a good idea for faculty to be putting in explicit language that what is allowed in this course, is not necessarily the same as what will be allowed in other courses, and it is incumbent on students to navigate those differences themselves.

Rebecca: And part of the reason why things might be different across courses is because the focus of those courses is different and really helping students understand that there’s reasons why policies might be different in other classes. It’s not necessarily arbitrary.

Don: Right, exactly.

Rebecca: So we always wrap up by asking: “What’s next?”

Don: Well, what’s next is I figure out how to do all this stuff, [LAUGHTER] and not just to talk about it.

Rebecca: …and you’re gonna send us a memo, right with that in it. [LAUGHTER]

Don: Oh yeah. Yeah, I’m happy. [LAUGHTER]

John: …or at least have ChatGPT generate a memo explaining….

Don: Exactly. So yeah, what’s next is to put this stuff into action. Of course, as I mentioned, some of the things here have already been experimented with, the non-ChatGPT parts of it at least, but really kind of integrating them and seeing if what I am imagining is what comes to fruition in terms of do these things dovetail as well as I think. I really think that they do. …that kind of pre-existing urge to go more towards the critical thinking element, and really, I think, does dovetail well with the AI, but putting it into practice, it will be over the course of probably all of next year. And so there’s going to be some experimental sections, most of the sections are probably not going to look very different than they did in the spring. And I think that’s probably a very good plan. But there’s going to be some experimenting in some of the sections at the undergraduate level, and part of a faculty learning community on problem-based learning. This course is going to be participating in that in the fall. And so a lot is going to come out of that, I think, as well.

John: Do you think there’ll be much buy in from other people teaching the course?

Don: So, students, by and large, do not like writing. Faculty, by and large, do not like grading writing. And so I don’t think this is one of those political monsters of how are we going to get this through? How are we going to make these changes work? I think there’s probably a lot of people who have nervousness about how you would make these changes. But with those two facts that I don’t think you’d get much disagreement from, I think even across disciplines, I don’t think it should be that difficult for this to be implemented

Rebecca: Well I hope you’ll join us after you’ve implemented some of the things to share some of your reflections and let us know how it went.

Don: I’m happy to.

John: Well, thank you, Don.

Don: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

303. Higher Ed Then and Now

Teaching practices have gradually evolved as we’ve learned more about how humans learn. From one year to the next, these changes may appear small, but the cumulative effect is profound. In this episode, Todd Zakrajsek joins us to reflect back on the changes that have occurred in higher ed during our careers.

Todd is an Associate Research Professor and Associate Director of a Faculty Development Fellowship at the University of North Carolina at Chapel Hill. He is also the director of four Lilly conferences on evidence-based teaching and learning. Todd is the author of many superb books, and has published four books in the past four years. His most recent book is a fifth edition of Teaching at it’s Best, a book he co-authored with Linda Nilson.

Show Notes

  • Zakrajsek, T. and Nilson, L. B. (2023). Teaching at its best: A research-based resource for college instructors. 5th edition. Jossey-Bass.
  • Zakrajsek, T. D. (2022). The new science of learning: how to learn in harmony with your brain. Routledge.
  • Harrington, C., Bowen, J. A., & Zakrajsek, T. D. (2017). Dynamic lecturing: Research-based strategies to enhance lecture effectiveness. Routledge.
  • EdPuzzle
  • PlayPosit
  • ChatGPT
  • Wayback Machine

Transcript

John: Teaching practices have gradually evolved as we’ve learned more about how humans learn. From one year to the next, these changes may appear small, but the cumulative effect is profound. In this episode, we reflect back on the changes that have occurred in higher ed during our careers.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Todd Zakrajsek, and I am with Todd here in Durham, North Carolina. Todd is an Associate Research Professor and Associate Director of a Faculty Development Fellowship at the University of North Carolina at Chapel Hill. He is also the director of four Lilly conferences on evidence-based teaching and learning. Todd is the author of many superb books, and has published four books in the past four years. His most recent book is a fifth edition of Teaching at it’s Best, a book he co-authored with Linda Nilson. Welcome back, Todd.

Todd: Well, thank you, John. Well, this is exciting. And Rebecca may be a long ways away, but I have never been arm’s length from a person who interviewed me for a podcast before.

Rebecca: Isn’t that cool?

John: And we’ve really done that before either at a conference or at Oswego,

Todd: I feel very special.

Rebecca: Well, we can celebrate with our teas. So, today’s teas are:… [LAUGHTER]

Todd: I’m drinking a peach mango that I got from some teas that John brought, which are fantastic.

Rebecca: John, how about you?

John: I am drinking a Tea Forte black currant tea, which I brought from Oswego, in a new mug that was given to me by Claire McNally, when she visited this area last week.

Todd: Love Claire, she’s fantastic.

John: And it has kangaroos on it.

Todd: Yeah.

Rebecca: And I can’t see it. Let me see it, John. Oh, that’s a cool mug.

Todd: It’s a good mug. I got a mug from her university. But I didn’t realize I should have brought it. So I feel bad about that. But it is a podcast. So I didn’t think about what it would look like.

John: That’s true, we generally don’t do a lot of visuals on here.

Rebecca: And I have a blue sapphire tea in my Tea Rex mug.

Todd: Well, that’s a nice mug,

John: We’ve invited you back to talk a little bit about how some of the changes you’ve observed in college teaching across your career have impacted how you teach today. When did your work in higher ed begin?

Todd: Actually, it started when I was a graduate student. So back in 1987. So there’s no reason to try to figure out how old I am. Now I’ve basically specifically dated myself here. I started teaching, I got to teach an introduction to statistics course. And I had so much fun that I taught again the following year. And by the time I left my graduate program, I had taught more courses in that program than any other graduate student had ever taught in the psychology department there. I really loved teaching right from the beginning, when from the beginning, very concerned about student learning, and just getting rolling.

Rebecca: What was it about the teaching, Todd, that really got you hooked?

Todd: Just watching the studentsis. it’s the same thing as it is today, when you have an individual who’s struggling with something, and suddenly they get it and you realize that they may eventually get it on their own, but you realize how much you’ve helped them to move that along very quickly. And facilitating the learning process, I just really love that. That doesn’t mean I was fantastic at it. But I really did love it.

Rebecca: Sometimes the things we love the most are things that we’re not great at to start with.

Todd: That’s true.

John: My experience was similar, actually, I started in 1980, with a course where I had a fellowship, so I didn’t have to teach. But there was a sudden shortage in the department. And they asked me to fill in. And I was planning to go on into research. But it was just so much fun teaching that I’ve never stopped.

Rebecca: I taught as a graduate student too, and taught the whole time I was there. But I started a little bit later in 2003.

Todd: Alright, so that was a couple of years later.

Rebecca: Just a couple.

Todd: Yeah, I had kind of a funny start, I will mention that when I first started that after the first semester of teaching, my students got almost all As and Bs. And the department chair called me in and he said, “I’m not going to have you teach any more courses.” And I said, “why not?” And he says, “Well, you give grades away like candy, we have to have better standards than that.” And I said, “Well, how are you basing that?” And he says, “Well, you know, we looked at the grade point averages.” And I said, “Well, how about if I bring in my final exam, and just walk through it, and then you can tell me how it could change to be more rigorous.” And so it was great. I showed it to him at the beginning. And like the bottom of the first page, the students had to calculate a statistical value, then I had them explain how they came about that number. But if they had used a different test how might it been inappropriately found and what the interpretation might have been, based on the fact that they had done it wrong with a different test. I thought it was important for them to understand how these things can change. The Chair said, “I can’t believe you have your students in the first class actually talk about various tests like that.” And I said, “Yeah, I did. Then we turned the page he says “You did nonparametric tests?” I said, “Well, yeah, we did parametric tests, but then I thought they should know the equivalent.” And he said, “We never do that.” And then he turned the last page and he said “You had them do a two-way ANOVA? You’re only supposed to go through one-way ANOVA.” I said, “Yeah, but we’d finished everything and we still had a week left. And I figured I might as well introduce the next concept to them. And so I showed them how to do a two-way ANOVA and they ended up with all As and Bs. So if you could help me in how to push their grades down and give them lower grades, I’m perfectly happy to do that.” And he then set me up with two courses the next semester, but it’s that reliance on the teaching evaluations is always funny.

Rebecca: Todd, it’s just funny, as we’ve gotten to know you through the podcast [LAUGHTER] it sounds so perfect that that was your first experience. [LAUGHTER]

Todd: Yeah, I’ve lived my entire career on the edge. [LAUGHTER]

John: And those sorts of arguments are still occurring in a lot of classes today about rigor and the need to keep grades lower.

Todd: Yeah.

John: They’re less severe than they were a few years ago.

Todd: Yes, but also looking at how well a person’s teaching based on student evaluations. I mean, we should be looking at authentic assessment. Some things have changed through the years, some things have not changed through the years.

Rebecca: Well, technology is one of those things that has changed.

Todd: Woosh, yeah.

Rebecca: Can you talk a little bit about what tech was like in the classroom when you first started and how it’s evolved a bit?

Todd: Yeah, I know you have some listeners who have been teaching for a very long time. So those of you have been teaching for like 30 to 40 years, just stop and think back about what it was like when we first started. For those of you who have been teaching like Rebecca since 2003, let’s just mention that technology back then was mostly pens and chalk and chalkboards. So back then, of course, there’s technology, there’s always technology, but we were using overhead projectors. This was long before the internet came along to really be used in the classes. LCD projectors were not out yet. Canvas, Blackboard, Sakai, all those learning management systems were not around. We didn’t have any of the ways to email individuals, you couldn’t email your students back then. And there was no ChatGPT to write your papers for you.

Rebecca: But there were calculators that could do all the work for you.

Todd: Yes, but this is the cool part. Back when I started teaching statistics, I’m glad you mentioned the calculators, huge debate back then was whether or not the students should calculate the statistical values by hand using the calculator, because computers had just come onto the scene and we could punch the data into a computer and have a computer run an ANOVA for you. Should you calculate it by hand? Should you run it to the computer? And there was a huge camp that said you should do it by hand or you will never understand a statistical value. And I said, “You know, we’ve got the technology there. Why don’t we have the students use the computer to do the mundane stuff, and we’ll have more time to talk about the theoretical and the important implications.” But even back then we were having the discussions about whether to use the technology at hand or not. Oh, and by the way, we are also hanging grades on doors. So we would figure out the grades, we’d tack it to the door, and then the students who want to know what their grades were for the class would swing by and look at the door.

John: And they were sorted alphabetically, to make it easier for people to find where they were in the grade list.

Todd: Yeah, it was great. We listed them according to their social security number, [LAUGHTER] which was a little different back then. And yeah, we actually did that back then. But as John pointed out, they were listed by number so nobody knew whose number went with whom, except, surprisingly, they were alphabetical on the door. So not only could you figure out Armstrong’s exam score, you’d get Armstrong’s social security number as well. Yeah, times have changed.

John: And it was also back in the day of dittos and mineos as well, which was the only way of disseminating information on paper.

Todd: This is so much fun. We’ll get to some real meat of this thing. But that walk down memory lane has some fun stuff too. The dittos…

Rebecca: I remember dittos, just for the record, okay.

Todd: Yes. So you probably remember, if you dittoed just before class, and you handed it out in class, the students would all pull the ditto up to their face, so they could smell the ditto fluid. And they got that smell. I was running dittos one time in the graduate student office, and I noticed when I looked down because it ran out of fluid, and I had to put some more fluid in, and I looked down and I noticed that the floor was kind of eaten away by this ditto fluid. And then… this is the best part… About a month later, I was digging for something in the closet and I found extra tiles and I thought they should put these tiles down to replace the ones that are all eaten and on the side of the box it said these tiles were long lasting and durable, reinforced with asbestos. So that ditto fluid was eating through asbestos tiles. That’s some strong stuff.

John: …to make it a little bit more friable so that it would disseminate in the air nicely.

Todd: Well, there had to be something to help the faculty members who were running all their own dittos to not mind doing it, and one way of doing this is to have them use ditto fluid, because I’ll tell you, you may not have liked it when you started, but by the end, it was all right. [LAUGHTER]

Rebecca: It’s funny that we’re taking this walk down memory lane, because on our campus, I was in our historic lecture classroom today in Sheldon Hall.

John: What are some of the other changes that have occurred and how have they influenced how we teach?

Todd: Yeah, so it’s interesting, I did the walk down memory lane and we were chatting about this stuff. It’s all fun, but thinking about how the changes have taken place. I think that’s really important. So there have been massive changes. I think that we tend to forget, it’s so easy to communicate with students now. Heck, people are texting now so that you can text back and forth with students. But think about how that has transcended or gone through time. There was a time when I would have to call and leave a message for a student on an answering machine, and then they would call back and we would try to find a time that we could talk on the phone. If we wanted to have a conversation. I could either leave a note for the student or I could call and leave a message that says least come see me after class. So even having a conversation with a student was difficult, then it became easier with email because you could start emailing back and forth. And now we have Zoom. And the equity in the way that this has changed, just think about the difference of this, if I’m leaving a message for a student, they may not even have an answering machine, if they’re living off campus with limited means back then. So even getting in touch with a student would be challenging. Now I can have a Zoom conversation with a student who doesn’t have to hire a babysitter, who doesn’t have to find reliable transportation, who doesn’t have to drive across town and burn gas, and to do all of those things that it would take to have a 15-minute conversation that in the past would have been really hard, and even four or five years ago would have been challenging. The grades, why in the world would a person have to leave… and I was teaching in very northern Michigan, there were days that the wind chill was 75 degrees below zero… and students would leave their dorm rooms and walk across campus to see a grade on the door. It’s actually physically dangerous. And now we have learning management systems, we could post things for students. Interlibrary loan used to take weeks to get a document that you can now go on and get. People can lament all of these technological changes at times, but we’re actually creating more and more equity within the higher education system as we make certain things easier. Not saying that we’re anywhere near an equitable system yet, but we’re moving in a really good direction. And a lot of those changes are helping us to get there.

Rebecca: I’m thinking about all the times when I get to go to the door or meet after class, it really assumes that students are a certain kind of student, they’re full time, they have time. And our students now are working [LAUGHTER], and where they’re juggling a lot of different schedules and things.

Todd: Yeah, and I mean, we want to be careful too. And I agree with you 100%. But they were juggling back then too. But some of the things we were doing, for instance, I taught a night class. Now I would probably suggest if I was going to teach a class from 7 to 10pm that I would teach it through zoom, because there’s a lot of reasons that it’s good to do. But I had students that I noticed in class, would very quickly at the end of class would start talking to other students and I couldn’t figure out what it was doing because a lot of buzzing and stuff. And what I found was that there were certain students who were uncomfortable, and we were in a very safe campus, but they were uncomfortable walking to their car at 10 o’clock at night. So I started saying to the students, “Hey, I’m gonna park a car… and when we showed up, there were quite a few cars there… but I’ll be under the second light, I drive a little red Chevette, not a Corvette, a Chevette, but I’ll have my car there. If you want to park near me, we can walk out together.” And there were students that were not paying attention to almost any of the class because they were fearful of how they were going to get to their car safely. When you think about Zoom and stuff, it’s even safety factors, I would never have a review session now like I used to at 8 to 9 pm the night before the exam because I’m exposing people to potentially dangerous situations. Now we’d have zoom sessions. But I could tell you 40 years ago, there was no even concept of what zoom would be and how it would work. Even Star Trek didn’t have stuff like that.

John: And there was also, besides the inequity associated with people who were working, many campuses had a lot of commuting students who could not easily get back to campus for office hours. Or if they were just taking classes on Monday, Wednesday, Friday and your office hours was on a Tuesday, they’d have to come in that extra day, arranging childcare, or their work to be able to fit that into the schedule.

Todd: Yeah, it really did start to change that system. So we got a little bit more equity, and like you were saying too, the commuting students, the part-time students, the students taking distance courses. When I first started teaching, I was writing… oh my word, remember the correspondence courses? …and you mail away and get a packet of material, you take a test at a local library and, and they talk about distance education being not as good as on campus, but at least better than nothing. And now we’re finally getting to a system where we can stop assuming that those folks who are coming in for part-time courses and stuff are just getting something better than nothing. They’re actually getting something similar to full college courses, which some of those online courses are actually as good or better than college courses that are on campus. But all that’s changing with the technology. It’s crazy.

John: And there’s a lot of research that supports that in terms of the relative learning gains with online and face-to-face, as well as hybrid courses, which seemed to outperform others in a few meta studies that have been done. But those were options that just weren’t available back then. And the early online courses were often designed to be replicas of face-to-face classes, and they probably didn’t work quite as well. But we’ve learned since that, which brings us to the issue of research. During the time that you’ve been teaching, there’s been a lot of research on teaching and learning. While some of it was taking place, it wasn’t very widely disseminated to faculty.

Todd: Yeah, that is true, too. It’s so much easier to get technology out. It’s easier to gather data, it’s easier to write it up. It’s easier to edit it so all of those types of things that are happening now that couldn’t happen before. And as a result, we’re learning a lot more about how people learn, you know, the book I did on the New Science of Learning, looking at a lot of the ways that students learn. And part of it’s just the ease of getting to information. But also part of it’s just being able to investigate how people process information. I used to teach Introductory Psychology back then, we would talk about the stages of sleep. And nobody really knew, for instance, what REM sleep was about, we knew that you had to have it or else it caused some problems. Deep sleep we knew was important, we now have indications that deep sleep for consolidation is necessary for semantic memory. If your sleep is interrupted, you can get eight hours of sleep. But if you don’t get deep sleep, the information doesn’t get consolidated. Procedural memory, how to give shots and kick balls and do anything procedurally looks like it’s more solidified during REM sleep. So again, the different types of sleep are associated with us learning long term, different types of information. We never knew that before all this technology was running around. In fact, back then I gotta say, I remember from my intro psych class being told that you were born with a certain number of neurons, and as you live through life, neurons would die. And if you killed them by drinking or doing something like drugs or something, they were gone forever, and you would never get more. And if you broke a connection, it was broken forever. That’s just simply not true. But it’s what we thought back then. So technology has really allowed us to look better at how people learn, different ways of helping them to learn and different ways they can even study. By the way, before we move on, we now have this physiological demonstration that staying up all night and cramming the night before the test. Even though it gets you slightly higher grades on the test, we now know that because the information is not consolidated that it won’t be there a week later or two weeks later. So we’ve always told students, you shouldn’t do it, but now we can actually show them why it doesn’t work.

John: And the LMS itself has offered a lot of ways of giving more rapid feedback to students with some automated grading with some things to give them more low-stakes testing opportunities. And those were things that we just couldn’t easily do back when you started teaching.

Todd: No, John, that’s a really good one. And we know that one of the most consistent findings right now in all of learning and memory stuff is that the more often you do something, the easier it becomes, long-term potentiation. Which means the more frequently you retrieve information from your long-term memory, the easier it is to retrieve. And just like you’d mentioned, we can now do LMS systems that are set up so that you could do practice quizzes, you could do dozens or hundreds of practice quizzes and keep pulling that information out over and over and over again. That was just not possible before this. And so the LMS helps with that, it helps by giving feedback, really good feedback so that students know what they’re doing well, and what they’re not doing well. And it helps faculty members to design feedback specifically for certain types of projects, and so that I can more easily give more feedback without spending a lot more time on it. So LMSs have done a tremendous amount of work. And that’s not even mentioning the fact that you can have all of the LMS systems loaded with the content. So students can log in and get their information without leaving their house. If there’s fiscal challenges with your class, you can put in articles, the students may not have to buy a book, they could read the articles. And so we’ve got students who were able to come to classes because they can afford to be there. By the way, I remember being on a committee when I was a graduate student, and we were looking at financial aid and different financial systems. And I remembered asking the Chief Financial Officer, I said, “What increase in tuition does there need to be before you start to see students drop off because they can’t afford to be here?” And this was about 40 years ago, but he said $100 for a year, if they have to pay $100 this year more than last year, some students won’t come back. If we look at the price of textbooks now, textbooks can cost $400. So, a book like that is definitely going to make a difference between some students being able to take the class or not. So LMS systems make this possible.

John: And they also make it easier to share OER resources that don’t have any cost for students, or some less expensive adaptive learning platforms, giving all students that first-day access. I remember, not so long ago, when I was still using textbooks in some classes, students would wait several weeks before they got that book. And that put them at a severe disadvantage. And the people who were being put at a disadvantage. were generally the students who came in with the weakest backgrounds because they came from lower resourced school districts.

Todd: Yeah, if they had the resources, they would have the better background foundational material, but they’d be able to buy the books. And you mentioned OERs. So open educational resources are really another thing that are really valuable because back then, before the technology, you couldn’t produce something that would be readily available like throughout the world. And so this project that’s going on now where they’re doing introductory level books in all the different disciplines, you can get an OER introductory psychology textbook that students can log in and read. None of that was possible before the technology. So even the creation of OERs has changed so much.

Rebecca: Well, speaking of digital materials, libraries have changed significantly too over time from having completely physical collections and interlibrary loans and things that take a lot of time to having a lot of digital resources, which changes access to research and materials that you can populate into your classes, but also can aid students in the work that they’re doing. Can you talk a little bit about the change in libraries and how that’s impacted how you’ve taught?

Todd: Yeah, you know, libraries have been fascinating to watch over the last 40 years, because it used to be the biggest challenge librarians had before them was which books to put on the shelves because there was a finite amount of shelf space. And there were lots and lots of books. And so that was the big thing. We used to take out journals that weren’t used very much to make room for other journals. Through time, little by little, they started digitizing all that stuff. And I can remember chatting with librarians, one conversation I had was back around 2001. I said, gonna be interesting, because there’s gonna come a day where there’ll be no books in the library, and the Dean of Libraries said “Well, there’s always going to be books.” I said, “Not always, potentially.” But even if we reduce them, I said, “What is your foresight? How is the library going to change?” And so he had a couple of ideas. But what it basically boiled down to our conversation is, I always felt like a library was like the brain of the campus, it had the books, and it had all of the information that you could go and get. As the books left, and things were diversified in a way that you could find this stuff, you could get all the information right from your dorm room, or from your apartment, when the internet came along, you could get anything you needed, then the library was still a physical space that was in the middle of campus. And what it should become is a learning commons, a place where people go to share and to learn from one another. And I think that’s what’s really changed is individuals still just pile into libraries and use the space, but they use it in different ways. They go there to meet other individuals to work, which they did before. But they took away that aspect of going there for the book part. And it meant all of those shelves got emptied, and they started pushing them out. And you can go into libraries right now that have very few shelves. But they have webcams, they have smartboards, they have spaces where folks can plug in their computers and share with one another. They’ve got screens set up so that you can project and have students sitting around a table, they’ve got Google Glass set up, all of these types of things that bring students together to use technology to learn from one another.

John: And they have cafes to help support that to make it easier for people to gather.

Todd: Yeah, you could swing by and get a cup of tea.

Rebecca: It’s funny, even when I was in high school, my sister and I would rely on going to the library to have access to a computer so that we could even type of paper, because we didn’t have one at home. And that kind of place of having the technology started a long time ago, but it’s amped up quite a bit over the last 20 years.

Todd: Yeah, and I agree completely. And the computers that are there. I mean, even right now, with the books dissipating, there a’re still large numbers of computers. And oftentimes, they’ll even be an area in a library that’s carved out with really high-end computers. But it gives students an opportunity to go. We make this assumption that everybody has a computer and they don’t. But libraries give them that opportunity.

John: Yeah, for those students working on smartphones or Chromebooks, that gives them access to all the tools that students with $2000 or $3000 or $4,000 computers.

Todd: Yes, because smartphones can work for lots of things. But they’re a little tough to write a paper on

John: When I started teaching, and probably when you did too, the predominant mode of instruction, which actually still is often the predominant mode of instruction in many departments, was lecture. That’s changed quite a bit since then. Could you talk a little bit about the shift from lecture-based courses to courses that involve much more active learning activity?

Todd: Yeah, or they just involve a lot more of everything. The concept of flipped classrooms, which was almost impossible 30, 40 years ago, because you really couldn’t get the information to the students. Yes, it was kind of possible, but whoo, if it was hard now, it was really hard back then. But the ability to get information out to students that they can read it before they come to class. But coming back to the lectures… So I’m going to take this moment and those of you who know me know that I’m going to do this, is that we still have no evidence that lectures are bad, but there’s something that we need to really keep in mind. I think this is vital. I do think it’s important for us to be able to talk about buzz groups and jigsaws and fish bowls and lectures and Socratic lectures, discussion lectures, all those different methodologies out there so that we know what we’re talking about when we chat with one another. But I do think it’s time that we stop talking about lectures being more effective than one thing or fishbowls being more effective than something else and look at the components of what is valuable in a learning experience.

John: And a good reference for that is a book on Dynamic Lecturing, which you happen to be a co-author of.

Todd: That is true and in fact that there’s the Dynamic Lecturing. And then there’s a chunk in that about The New Science of Learning. And then there’s a whole chapter in that about Teaching at its Best, because that’s a good point, John, thank you.

Rebecca: It’s almost like you’re trying to slip it in everywhere you are.

Todd: Because the research… people keep talking about one methodology being better than another. Here it is, folks, you can be a hideous lecturer, you can be a phenomenal lecturer. And if you’re a hideous lecturer, you’re not going to learn anything. If you’re a phenomenal lecturer, students will learn from you but they won’t learn all the time, it depends on some student factors. I’ve actually been exposed to group work in flipped classrooms that were awful. And so that concept is we start thinking about and this is why it’s going to come back to the technology, we think about the elements that need to be there, that are necessary for learning to take place. I’m just going to do this, because it’s not the topic I’ll make it very brief, is let’s just go with three things. If you don’t have your attention, as a teacher, if my learners aren’t attending to what I’m saying, if they’re on their phone or thinking about bacon, then they can’t process what I’m presenting. And if you’re having a think-pair-share, if they’re not attending to the person they’re sitting next to, you have to have attention. Number two, they have to have some value. If I’m hearing somebody or I’m reading something, and this has no value to me, it’s really hard to get it into your long-term memory and to learn it. And number three, I have to have a clue of what’s happening, I got to understand some aspects. Now if we think about attention, value, and understanding, now we can flip back to the technology. This is why gaming works. Gaming draws the attention, it increases the value, because you want to win the game, and it has understanding. We have all played games. You open up the old board games, and now it’s digital, where you don’t have a clue what the game is. It’s like, if you advance a player four pieces and the opponent advances five pieces, you have to go back three spaces, unless it’s a Tuesday. When those instructions are that complicated, you don’t understand. So we can use technology to help with attention, we can use technology to help with the value of what’s going on. And we can use that technology to help with understanding. Those are things that were very difficult before. And they allow us to do things like a mini lecture and then shift over to an active learning exercise, and then say, take all this information and create a Zoom session tomorrow that will go over it again. So the technology has really helped us to be able to do all of these things to get at the core of learning, a topic I barely care about. [LAUGHTER]

John: That’s an important one, because people often see this as this binary issue where you lecture or you use active learning. And there are some really effective ways to combine them. And in fact, in that book on dynamic lecturing, it was suggested that lecture can be more important in introductory courses, when students don’t have as much of a knowledge base.

Todd: You’re absolutely right. Discovery learning is a really great way to learn if you’ve got a lot of time. I can just put you into a room with some other people and say, “Here’s some data, and here’s some things we need to know. Go.” And if you don’t have any foundational knowledge at all, it takes forever to figure it out, you go online, you know what to look for, I could do a five-minute lecture, and at the end of five minutes, set it up and say, “Now go and work with your neighbors. In fact, here’s what we’re going to do, we’re going to have you each work in small groups in class, I’m going to open up a Padlet. At each table, I want you to go in and add your information or put it into the column that corresponds with your group number.” As an instructor, I can watch everything develop in front of me. While I’m in the room, I can look at my laptop and see it and walk over to a table and say, “looks like you’re struggling a little bit.” I’ve lectured, I put them into small groups, I’ve had them use technology, I’ve created a little bit of competition on who can come up with what and I’ve had a way for me to monitor it and give them feedback. That is so different than what teaching used to look like. So pulling it all together, that’s what we do.

Rebecca: The tools to be able to monitor have been really helpful in my own teaching and being able to get a better pulse on what’s going on and get a nice overview and then be more targeted in how to interact with small groups rather than just kind of wandering around more aimlessly like I think I did initially. [LAUGHTER]

Todd: Yeah, and this is all going to be great until we get our cognitive load headbands that I’m waiting to be developed. So anybody who’s listening, take this idea, run with it, you can make a bazillion dollars and then take me out to dinner or something. I want a headband and the headband has a light and it measures brainwave activity. And then as I’m teaching, if you start to be a little bit like it’s a little bit too much, you’re moving out of that zone of proximal development, the light turns from green to a yellow. And then when it hits red is like when you’re trying to put together an Ikea bookcase and someone comes by and says “What do you think of this?” and you say, “Errr, I’m working on an Ikea bookcase right now.” …that shutting down with that red light. I’m telling ya, that’s going to be the technology we’ll want next.

Rebecca: It would be so helpful. [LAUGHTER]

Todd: You can actually look and see somebody else’s zones of proximal development and their cognitive load. Whoof. Which by the way, there’s a little party game that they’ll do periodically at parties. It’s like if you’re a superhero, what would you want your superpower to be? And I was in a room one time and one person said they wanted to fly and somebody else said that they wanted to be invisible, which real quickly in my head, I’m thinking, what could you possibly gain that wasn’t illegal or creepy if you’re invisible. So aside from that, transporting and everything else, and they got to me, and I said, “I want to be able to see people’s zones of proximal development. If that were my superpower, I’d be the best teacher.”

John: I bet that went over really well at those parties. [LAUGHTER]

Todd: Yeah, my friends all said “You are amazingly smart and quite insightful.” They used different words, but that’s what I heard. [LAUGHTER]

Rebecca: They didn’t start with what is that? [LAUGHTER]

Todd: As soon as I start talking, most of my friends just shake their head and drink whatever beverage they have near them. [LAUGHTER] So yeah, it’s good times, good times. They’re all impressed. They don’t say it all the time, but I know they are.

Rebecca: I think one of the things that often happens with technology is that it allows us to get things quickly and move through things quickly. But sometimes, as you just noted, learning doesn’t happen quickly.

Todd: Yeah.

Rebecca: Can you talk a little bit about speed and the difference between maybe not having all the technology and all the things really quick versus maybe now where we have it at our fingertips, but do we always want it at that speed?

Todd: So there’s another study that I’m waiting to see. This is an easy study, folks, somebody can run this one quickly. We all know that students are listening to any recorded lectures or recorded material that they have to watch, 1.7 is about the best speed that we tend to see people listening. 2.0 is a little bit fast for some folks. 1.0 is like normal speed, that’s no good, too slow. So what I’m curious about is the space between words and between sentences that our brains, because they move so fast, we can listen faster than somebody can talk. And we have all this other stuff going on is I can be thinking and processing while you’re talking to me. But if I bumped that up to one seven, I think we close the gaps. And I hear it a lot faster. But what I don’t think is happening is the cognitive processing while I’m listening. The active listening component to it. So I think technology can create concerns in those directions. And students who do try to process material too fast… we’ll wait and see.

John: And that’s especially important in flipped classrooms where students do watch these videos outside. One of the things I’ve been doing with those, though, is embedding questions in the video so they can watch them as quickly as they want. But then they get these knowledge checks every few minutes. And then if they find they’re not able to answer it, they may go back and get their attention back and watch that portion again.

Todd: Yeah, I think that’s a really good way to go. EdPuzzle’s kind of a fun technology to use. I don’t know if that’s the one you used.

John: I’m using Playposit, which is a bit more expensive, it works beautifully. I love it, they did just double the price this year, though, it was bought by a new company.

Todd: This is the tricky spot now as the prices are going up. You know, inflation is a terrible thing to waste. Anytime somebody can raise prices now it’s like, “ooh, inflation”. So you know, prices double, inflation is 8% with runaway, now it’s back down around three. But when inflation was 8%, they doubled the price and say, “Hey, we’ve got to,” but yes, it’s some of them are expensive. There’s lots of things that are less expensive. Oftentimes we pay for functionality that help us but the freemiums kind of thing. So stuff that’s inexpensive. I just wanna let everybody out there know just about anything you want to do in class or can think about doing it, there’s a way to do it for either free, or probably under $100 a year, which I know $100 can be expensive for some people, it’s about eight bucks a month. And so things like Padlet that I think might be up around 140 now, so maybe $12 a month, can change how much time you spend doing things, and how much time for students. But yeah, I love the embedded questions to help slow things down.

Rebecca: I think that the cognitive load can happen really quickly if we’re piling lots of information in but not always providing the time to process and use that information in some way in the kind of activities that you were talking about. Or knowing when everybody’s red light is going off in the class.

Todd: Or when people try to do multiple things. I mean, now you’ve got the technology around. So if students are trying to listen to an assignment while they’re texting their friend and have a TV on, I mean, we’re living in an age where there is a lot going on, and people believe they can process lots of things. Evolution doesn’t happen quite that fast. And so I think we have to be careful with that one.

John: One other thing that’s happened is back when you and I both started teaching, the only way students generally communicated their learning was either on typed pages or on handwritten notes. Now we have many more types of media that students can use. And also we’ve seen a bit of an expansion of open pedagogy. How does that help students or how does that affect student learning?

Todd: Wow, that’s really changed a lot as well. Blue Books. Remember the blue books? I think they still sell blue books in the library. They may cost more than the I think it was eight cents when I started, but the concept of writing things down, you turn them into the faculty member, the faculty member would grade them and turn them back. One of the big things that I caught years and years ago was so much wasted cognitive energy in terms of what they produced. I’d read a paper from a student and think this is amazing, and no one will ever see it. It was written for me, I graded it. And now it’s done. I think the technology has changed so many things. One of the biggest things, I would encourage all the listeners, any faculty member out there is, whenever possible, create something that will take the students’ work, the things that they’re doing, and use it to make society better. It’s not that hard. There’s assignments that you can do on Wikipedia. Anybody who wants to complain about Wikipedia, if you don’t like it, I’m gonna go back to Tim Sawyer, who is a faculty member of mine, my very first time I ever did TA work. I was complaining about some students. And he said, “You can complain three times. And after you’ve complained three times, either stop talking,” he was a little bit ruder about that, “or do something about it… just shut up or do something.” And so I complained about Wikipedia for a while, that it wasn’t all that effective. And I thought, well, if I don’t like the page on cognitive load on Wikipedia, I could give an assignment of my cognitive Psych class to go on to Wikipedia and fix it. And so you can have Wikipedia assignments, there’s so many things you could do. Here’s one for you. If you’re doing one on communication, you could have your students go and take pictures or short videos somewhere on campus of something that’s meaningful to them, and then jot down why it’s meaningful, take that compilation of stuff and send it over to the office on campus that does publicity. What better way of drawing students to campus than to have all of these students that have said, I love sitting by the pond because… and in the past, we would have had students write a paper about someplace on campus that you think is effective, put it in the blue book, we would grade it, we turn it back to the students. And that is a waste of possibilities. And so I think we do have lots of ways that we can get the students involved in helping through technology,

John: One of our colleagues in SUNY, Kathleen Gradel, had an assignment for a first-year course, where the students went out, took pictures, geocoded it and added it to a map layer that was then shared with other first-year students about useful resources on campus and their favorite spots on campus, which is another great example of that type of authentic learning.

Todd: Yes, for the authentic learning, there are just so many possibilities because of the technology. If anyone doesn’t have ideas, ask deans, ask the provost, ask the president on your campus, like what kind of information would be helpful, either for the next round of accreditation or for just helping the campus and we can design those things. Another one I did was we took students to the museum. We’d go to the museum, almost any class could kind of find some way to tie museums in, and through the museum, not only would they write stuff that the folks at the museum who did curation would help use, but also just helping the students to see how issues from the museum, how artifacts and things can be used in their own life, to better understand.

Rebecca: When I first started teaching, community-based learning was popular, in fad at the time, and I think having the experience of being a student in a class like that, but then also a faculty member teaching classes like that has really informed the kinds of projects that I do. Maybe they’re not always community-based learning, but they’re often community oriented, whether it’s the campus or even the surrounding community that the campus is situated in to help students get connected. There’s so many nonprofits that need partners and love, there’s always a project that can be done. [LAUGHTER]

Todd: There is. And I used to be a director for a service learning component of the campus. And yeah, there’s just so much out there that we can do to help others.

Rebecca: And students always had such a strong connection. And they didn’t want to fail because other people were depending on them. And so there was a real investment in the work that they did on projects like that.

Todd: I will admit that I’ve never experienced it myself. I’ve never even heard of anybody that if the students are doing some kind of authentic learning, that their authentic learning is then used to help somebody else. I have never heard students say “What a waste of time” or “I hate that class,” or “those assignments are just busy work.” They’ve never used those terms.

John: One common sort of project is to create resources that could be shared with elementary or secondary school students in the disciplines. And again, they can see the intrinsic value of that.

Todd: Yeah. Students could write short manuals on how to learn and then pass that on to the first-year students. And so upper-division students could be helping the lower-division students because not everybody can get a copy of The New Science of Learning, third edition.

John: …available from… [LAUGHTER]

Todd: Available at… used to be Stylus. Since Stylus was sold to Routledge, now it’s available at Routledge. [LAUGHTER]

Rebecca: Given the historical background that we’ve walked through today, what if we think about the future? Where do you see technological changes or learning theory changes impacting the future of higher ed?

Todd: Yeah, we’re living at an interesting time. I like to point out to folks that when you go back to Socrates, Plato kind of time there was a thought that if you wrote something down, it would weaken the mind so we shouldn’t write things down. Luckily, some individuals wrote things down or we never would have known. We’ve gone through several iterations of those kinds of things. Samuel Johnson, I believe it was, who said “With the ready availability of books, teachers are no longer needed. If you want to learn something, you could go get a book on it.” Well, that was a couple of 100 years ago. And we still have faculty members, we have students writing things down, we’re reading, I don’t imagine how you could teach without writing things down and having books. The internet came along, as we were discussing earlier, while we were teaching, we watched the internet show up. And there were people who said, “Well, with the internet, there’s going to be no need for teachers anymore, because students can get whatever they want.” I can’t imagine teaching without the internet right now. So as we’ve gone through each of these iterations, there’s been this fear that maybe we’d be supplanted by some technology followed by “I don’t know how I’d work without that,” it’s a little trickier now, because with generative AI, we’re talking about not just something being available, but actually creating something. I don’t know what that’s going to look like. But there’s some real possibilities that the generative AI ChatGPT, could do things like help students who have writer’s block, get started. And that’s an individual that maybe could produce something really cool, but just can’t get started. I didn’t publish my first book until about seven or eight years ago, because I’m one of those individuals who has a terrible time from a blank screen. I just have a terrible time with that. And so now, I don’t use ChatGPT to actually write anything significant. But I will tell you that I will use it for the first paragraph. That’s all, just one paragraph. And then I completely rewrite that. And there’s no actually trace of it. But it’s something that gets me going.

John: So can we count on more than a book a year going forward? [LAUGHTER]

Todd: No, no, no, no, you can’t. So exhausting. But the concepts that will help students that can do that, I think that’s going to be helpful for them. So there’ll be a type of student who couldn’t have produced before, but now can. We are definitely going to run into some challenges, though, with students who are going to just use generative AI and use artificial intelligence to actually create and to hand something in instead of doing the work. So I do think we’re in a challenging time right now. And I wouldn’t make light of that. There’s actually something that I find fascinating from this. Right now, more than ever before, we can actually have artificial intelligence create something for us, especially in higher education, this hasn’t been done before. The tricky thing is that we were the ones to be able to make that possible, because we learned things. If we let a machine do that work for us, we’re not going to be put into the situation or our students coming along, will not be put into a situation where they’re intelligent enough to do the things that need to be done when they need to be done. And so I do think we’re facing a real dilemma right now. If my students, for instance, always do use some artificial intelligence to create a paper and hand it in, if I can’t catch it, they may end up with an A in that portion of the class. But there’s going to come a day when they’re going to have to write something or be able to read something and tell if it’s written well. And so I’m a little bit nervous, we’re entering a phase where by bypassing some cognitive processing that needs to be done, we may be limiting what we’re able to do in the future. Wrapping this up, though, I don’t want to be the person who says if you use a calculator, you’ll never understand this statistical test. So I don’t know where the balance is. But I do think we’re going to have to have decisions coming up that we’ve never had before.

John: Generative AI is drawing on that wealth of knowledge that has been produced. And for that to continue to grow in the future, we do need to have some new materials being created. So that is an interesting challenge, unless it goes beyond unless….

Todd: …unless it creates it. So that was one I thought about, by the way, sometimes you’re sitting around just thinking about stuff and it’s interesting. I was thinking how do I acquire new information. And the way I acquire new information is I go read articles, I read books, I read a ton of stuff. And then I say I think this is valuable, I don’t think that’s valuable. And then I put it together and say here’s what I’m thinking. And now I’m looking at this generative AI who goes out and scans the environment and pulls these things and then creates something new. It doesn’t have the cognitive processing that I have at this point, but…

John: it’s in the early stages.

Todd: We have some folks who are very concerned out there, especially in European countries that are starting to put some guardrails out, because at the point that it keeps grabbing stuff, and then generating and then it grabs the stuff it generated, then it’s going to be interesting. But as of right now, I just read another article, I think it was yesterday, that they’re going out and grabbing the most popular or most frequently written things and then putting it down as if that is right.

Rebecca: The way that you might prioritize as a human with an expertise in something, is going to be really different than a system that’s prioritizing based on popularity, [LAUGHTER] or like how current something is like when it was last published. That’s a really different value system that really changes priorities.

Todd: Yeah, and I think it changes how we teach. I think the way we teach is going to fundamentally shift because we’re going to have to work with students with all these things being available and explain to them and talk to them about the learning process and the value of the learning process. And keep in mind, this isn’t just about ChatGPT writing papers, everybody’s freaked out about that right now. We shouldn’t lose sight of the fact that you could get fresh, cleanly written papers that have not been plagiarized at all, we’d be able to do that for 20 years. There are paper mills, I can either write away or contact somebody and say, “Please write me a 10 page paper on Descarte, and they would write it, I could turn that in. What actually has happened recently is that everybody can do it, even those who can’t afford to have a paper written at $10 a page or whatever it’s costing. And so equity comes back again. [LAUGHTER] Now we’re an equal opportunity cheater. So we have to be careful with that. But I think the way we teach is going to change because all that information is going to be available, kind of like the internet on hyperspeed. And then what do you do with that? It’s going to be really intriguing. I think it’s an exciting time.

Rebecca: So Todd, this episode’s gonna come out right at the beginning of this semester. So you’re saying we need to be thinking about how to change our teaching. ChatGPT’s here, what are you doing for the Fall differently?

Todd: Well, I think the biggest thing is what we were just talking about, looking more at the learning process, which has been a big thing for me for the longest time, is explaining and talking through the learning process, I can hand you all this information, but if I hand it to you, you don’t learn. In fact, one of my favorite examples came from a friend of mine, and it was the gym, if you want to get in better shape, I could pay somebody to go do sit ups for me. And then I could somehow log in the book at the gym that 100 Sit ups were done, use the passive voice there, and somebody else did them for me, I’m not going to get in better shape unless I do the situps. So I have to do the work, I have to run, lift weights, do the situps in order for me to be able to gain. We need to just turn that into a cognitive process for our students to really gain cognitively they have to do the work. And so I think more than ever, it’s how do we convince students of that? And for the faculty members who say, “Well, that’d be great, but my students just want the grade.” If that’s the case, we have a bigger problem than whether or not some technology can write a paper for them.

John: So how do we convince students that it is important for them to acquire the skills that we hope they get out of college?

Todd: I think this is probably going to come down to the community building, it’s been there forever. If you really want your students to do the work, the best thing you can do in my view, and that’s why I’m gonna say, Rebecca, I don’t think a lot for the way I teach, has changed. You build a community, you build relationships, you talk to the students about importance of things, if you’re sincere about that, and they get that then yes, there’s going to be some students that are going to mess with the system, they have always been there. But you’re also going to get a lot of students who will say, “Yeah, that’s a good point.” And then they’ll do the work. I don’t teach as many undergraduates as I used to, I’m teaching more faculty than ever because of being the faculty developer. But there were years that I would have to tell my students don’t put more time than this in on your paper, you have other classes, you need to do the work in the other classes. Because, and I’m telling you, I am very proud of this, my students would spend a ton of time on this stuff for my class, because they didn’t want to let me down. And I would say you’ve already got an A, I’m proud of what you’re doing, please go work on your other classes. That kind of scenario happens when you build community. And I’m not saying it’s easy, I would never say it’s easy. And it’s not going to happen for everybody. But it is the foundation of good teaching.

Rebecca: So we always wrap up by asking, what’s next?

Todd: There’s just so much going on right now. I think that what’s next for me is I am still in that headspace of coming kind of back from the pandemic, anybody who says, “Yeah, but the pandemic’s all over,” wait for November, we won’t know, we’re going to see. But I still think that’s next is kind of thinking about how we teach and learn in this environment. So moving in that space, it’s probably not surprising. I’m working on the next book here. One of the things I want to do now is the last couple of books that I’ve done had been pretty heavy books. And now I want to write something that’s a little bit lighter. So it’s going to be more of a quick guide with more narrative and having some fun, I love telling stories. I love having fun with people. So I’m going to try to create a book that’s kind of like a science of learning and teaching at its best but really accessible and more of a story-based kind of way of looking at things.

Rebecca: Who is your audience for that book?

Todd: Anybody who will read it? [LAUGHTER] Anytime I write anything, I have to have the audience firmly in mind and think about who am I talking to. And I really believe there is a pretty big overlap with students and faculty who don’t know specific things. And I’m not saying this in a mean way toward any of my faculty colleagues at all. But there’s a lot of people who aren’t taught about things like long-term potentiation and deep sleep in terms of semantic memory, and looking at depth of processing and those types of things. So the same type of thing we can say to a student, we know you shouldn’t cram, but here’s why you shouldn’t cram… faculty learn a lot from that as well. And so my audience for this book is going to be faculty and students, students, because I think it’ll be more fun to read about how to learn in a narrative form like that. And faculty because it’s more fun to learn when you read in that kind of a format for some people. we’ll see.

John: And if faculty design their courses to take advantage of what we know about learning, it can facilitate more learning.

Todd: Wouldn’t that be cool? We could just keep rolling, rolling. What a great amount of work. I mean, a huge amount of work that faculty do. They’re hard working folks that are just cranking away all the time. Number one, making their life a little bit easier by helping to understand things would be great. And just having a little bit more fun would be fun, would be nice way to go to0.

Rebecca: Hey, anytime you can save time, so that we can have more play in our lives is better.

Todd: Yeah, just to do whatever you want to do.

John: Yeah, ending on a note of fun is probably a great way to end this.

Rebecca: Well. It’s always great talking to you, Todd. Thanks for chatting with us and going on the Wayback Machine.

Todd: Oh, you know, I love the Wayback Machine.

Rebecca: I love it too.

Todd: For those of you who don’t know about that, you should check out the Wayback Machine

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

302. Flipped Team-Based Learning

Flipped classrooms allow for class time to be used to put content into action. In this episode, Tina Abbate joins us to discuss the team-based approach that she uses in her classes to help develop the real-world skills important in her field.

Tina is a Clinical Assistant Professor at Stony Brook University’s School of Nursing. She holds a collection of credentials including a PhD, MPA, an MS, and is a registered nurse (RN). She teaches in-person and online undergraduate nursing classes at Stony Brook and conducts research on active learning strategies and the retention of information. She works as a nursing supervisor at two local hospitals.  She is the recipient of the 2023 SUNY FACT2 Award for Excellence in Instruction and was a recipient of the Stony Brook University Award for Excellence in Teaching an In-Person Course.

Show Notes

Transcript

John: Flipped classrooms allow for class time to be used to put content into action. In this episode, we look at one instructor’s team-based approach that emphasizes real-world skills important to the field.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist….

John: ….and Rebecca Mushtare, a graphic designer….

Rebecca: ….and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Tina Abbate. Tina is a Clinical Assistant Professor at Stony Brook University’s School of Nursing. She holds a collection of credentials including a PhD, MPA, an MS, and is a registered nurse (RN). She teaches in-person and online undergraduate nursing classes at Stony Brook and conducts research on active learning strategies and the retention of information. She works as a nursing supervisor at two local hospitals. She is the recipient of the 2023 SUNY FACT2 Award for Excellence in Instruction and was a recipient of the Stony Brook University Award for Excellence in Teaching an In-Person Course. Welcome, Tina.

Tina: Thank you. Thank you so much for having me here today.

John: We’re very happy to see you again. We saw you at the SUNY Conference on Instruction and Technology (or CIT) about a month or so ago. And our teas today are:…. Tina, are you drinking tea?

Tina: I am. I am drinking a chai tea. Very good.

Rebecca: That sounds nice and warming.

Tina: Yup.

Rebecca: It’s a little chilly here, although it’s summer and it was hot yesterday. It is not hot today. [LAUGHTER]

Tina: Yes, for sure the weather has been very odd.

Rebecca: So I have my tea for teaching mug today. And in it, I think actually a mix of a couple of different black teas because I switched when I had a half a cup left. [LAUGHTER] I’m not sure what we call this today, but it’s a mix of black teas.

Tina: That sounds delicious.

John: Well, it sounds like a great tea to have while discussing blended learning.

Rebecca: A high quality blend. [LAUGHTER]

John: And we’re having a real cold spell here in Durham, North Carolina. The temperature has dropped down to 87 today, and I am drinking a tea forte black currant tea.

Rebecca: That’s a nice summer tea.

John: It is.

Rebecca: So we’ve invited you here today to discuss your use of active learning tools. But before we jump into that, we were curious about your wide range of degrees, credentials, and certifications. We didn’t even list them all. Can you share a little bit about your pathway into your current position at Stony Brook?

Tina: Sure. Well, when I went back to grad school, I certainly didn’t intend to get three graduate degrees. I had gotten into Binghamton and gotten into their BS to PhD program because I wanted to do research and my ultimate goal was to do executive leadership position at a hospital because I really enjoyed the leadership role of nursing. So just to backtrack, I graduated Binghamton University in the year 2000 and started right in the NICU (neonatal ICU) at Stony Brook and I worked as a NICU nurse for six years. And in that time, I knew that I wanted to go back to school. And like I said, I got into the BS to Ph. D. program at Binghamton. They awarded me a fellowship program. So I moved from Long Island. My daughter was one at the time. And I started my education there at Binghamton, continued it for the graduate program. And about a year into my doctoral studies they had asked if I wanted to teach clinical and I’ve taught in other capacities. I used to teach violin and piano when I was younger and I never really thought of teaching as a career goal for me. However, I was a poor graduate student, and I said, “Sure, I’ll do it.” And I had about six students in the NICU. I was teaching clinical, and, I don’t know, something came over me. I found my professional soulmate, something clicked so hard for me in that clinical that I wanted more. So I continued asking for teaching assignments. And it’s hard to articulate the feeling that you have, but I felt like I found my niche. And so I did clinical instructoring for about six years and then I moved into the classroom setting. So at that time, I still worked as a nursing supervisor, so I enjoyed the leadership role. And Binghamton started a dual master’s degree program, where you get your master’s in nursing with a concentration in whatever you wanted, I chose education. And the other part of the dual degree was a Master of Public Administration. So I was the first cohort to move through that program. So I graduated first with my Masters of Science in Nursing and my functional role was educator. Then two years later, I completed the Master of Public Administration, and then eventually the PhD. And it all just aligned so perfectly in my current career, because obviously I’m an academic at heart through and through. So those degrees have assisted me in that role. I still work in administration. I teach research, I teach leadership and management. So all of the degrees I’ve utilized and I still utilize actively every day. So this pathway was kind of carved out for me, I think, and I just feel very fortunate that I’m able to apply all of the degrees that I’ve gone for.

John: At the SUNY Conference on Instruction and Technology, you gave a presentation on how you structure your courses. And you mentioned that you were using a flipped team-based learning class structure. Can you tell us a little bit about how your classes are structured, and what a typical class day would look like in one of your classes?

Tina: Sure. So any class that I’m involved in or coordinate, the structure that I utilize is a flipped team-based learning approach. And this essentially requires students to prepare prior to coming to class. It has some benefits there, there’s flexibility, students can learn at their own pace, it really amps up the student responsibility for learning, as we know, and then it also gives us the opportunity for higher level learning because they’re interacting with the concepts outside prior to class. And the team-based part of it I like is because that increases that collaboration amongst students. We know that nursing healthcare is a team sport, so I like to engage the students in teamwork so that they can collaborate and work on their team dynamics, and their own personal team skills. So how my classes operate is, prior to each class, students complete a set of videos, and they’re interactive videos, they’re accessible videos for all types of learners, and it carries weight in their grade. So basically, in these pre-class videos, students get a little voiceover content from me about a concept, and then they get tested on it using a variety of types of questions: matching, true-false, multiple choice, hotspots, you name it. As they move through the videos, they are taking notes on a note-taking guide. So all the concepts are there for them to just follow along, take notes. So they’re seeing, hearing, they’re doing something as they move through the videos. And that note-taking guide eventually acts as a study guide for them, because they have to take a quiz every single class. So they complete these videos before class. And then I start each class with a micro-lecture review using Kahoot!, which is just a game-based learning platform. And in this micro-lecture review, I’m really drilling down to the concepts and helping these students reconcile any last residual confusion that they may have about these concepts. And then after the Kahoot!, they take a quiz. Now, since they’ve interacted with the concepts so many times prior to taking this quiz, I push the level of the questions in these quizzes. There are 15 questions and I try to push the level as high as I can. And the students are able to rise to the occasion because they are not hearing the information for the first time when they walk into class. They have a vague sense of the concepts, we nail it down, and then they take the quiz. After the quiz, the rest of class is comprised of team-based activities. And that’s how every class looks like for me.

Rebecca: Can you talk a little bit more about the embedded questions that you have in the videos and how students have responded to that aspect of a flipped classroom?

Tina: Absolutely, I use a program called Articulate 360. Articulate 360 has many different types of functions in it. But I focus more on the storyline aspect of this product, where I’m able to set up these video clips. So if you already have voice overs, you can basically chop up that voice over into different bits, put it into a story, the type of file that they reference there. And then in between each clip, you can embed any type of quiz question that you could possibly imagine. And you can set up different parameters. So for example, I like to elevate the stakes a little bit, so the students, for these pre-class videos, the grade that counts is their first pass. So it’s not like they can retake the video for a higher grade. It’s whatever they get at the end of that first pass of the video is the grade that counts. And they have two opportunities to answer each quiz question correctly. And I also embed a lot of feedback, so if they get the answer wrong, they’ll see a pop up with some review, and then if they still got it wrong, or they got it right, then there’s an explanation that pops up for the right answer. So I do survey my students in the middle of the semester using a Google form. And then at the end using the university platform, and the feedback about the videos has been very positive, they really do appreciate even though it means extra work, I’m still not giving them 20 chapters to read. I’m giving them something that passes along a bit more quickly and has a better chance of sticking in their memories. And they also appreciate the note-taking guide because it also becomes a study guide, not just for the quiz, but for the final exam at the end.

Rebecca: Like I’ve counted four or five layers of countability on that same content. [LAUGHTER]

Tina: Exactly.

Rebecca: We’ve got the note taking guide. We’ve got the embedded questions, and we’ve got the Kahoot!, and then we’ve got the quiz, and then the exam at the end.

Tina: Yeah, so it’s all about building on these concepts, having the knowledge and then being able to apply it in the classroom

John: In your presentation, you mentioned that you were de-identifying the names of students taking the Kahoot!, but maintaining a leaderboard in the classroom. Could you tell us a little bit about how that works.

Tina: So Kahoot is based on answering the questions correctly or incorrectly. And part of the score is how quickly you answer the question. So ideally, you want to answer quickly and answer the questions correctly. So at the end of the Kahoot!, they get a score. And just again, to raise the stakes, students have to hit a certain benchmark of points to receive full credit. And I try to push that benchmark a little bit, not to make it impossible, but just to make it a little bit challenging for them to give them something to work towards. So for example, in one semester, they have to reach 70,000 points to get the full credit, and then it’s prorated from there. So every time I have a class, I load the data into this program that was built by our Center for Excellence in Learning and Teaching on campus. One of the computer scientist was able to put this leaderboard, showed me how to upload the files, which are basically just CSV files. And what it does is this leaderboard shows their rank in the class, their total score, and the score for that week, so that they can monitor their progress. And everybody else is de-identified and random words, but they can see their name, and they could see their rank in the class.

John: And one of the advantages, I think, of using Kahoot! is it does provide some practice in developing automaticity. So that students can practice retrieving information quickly, which I would think would be especially important in health-care situations.

Tina: Absolutely. And I’ll have some students that come to me and they just absolutely despise Kahoot! because of the stress. And if you’ve ever taken a Kahoot!, and I have, it is stressful, you have to really think on your feet very quickly, especially since your score is based on how fast you answer the question. So what I tell them from the beginning is if you really are struggling with Kahoot!, and you don’t like Kahoot!, Kahoot is really for you, it’s meant for you, because I want you to think of a situation in a hospital setting. If a patient is deteriorating, we call something called a rapid response. And a team of people flow to the room to address whatever issue it is, perhaps the patient’s having difficulty breathing, whatever, chest pain, this now has become a very emergent situation. And in that situation, you have to be, as the primary nurse or a nurse assisting someone else, you have to have laser focus, and someone may ask you to just go get a piece of gauze. And if you’re new in the role you may be so flustered, just by getting that piece of gauze. So, this is really like a precursor to that. So I tell the students to use Kahoot! as a mechanism to help with your laser focus in situations where the outcome is dependent on what you’re doing.

Rebecca: Another thing that seems really relevant to a healthcare setting is the team-based learning aspect of your course. Can you talk a little bit about how you arrange the team-based activities and also how you set your students up for success on teams.

Tina: So with team-based learning, as we know, it’s simply a collaborative learning strategy and how the team activities look depends on the course. So I can talk to you about my research course. That happens every fall semester, and I have 160 students, this is the graduating class. These are the seniors, they’re in the last two semesters of the program. And what we do in that course is the team-based activity portion of class is working on a project. So I’ll tell you a little bit about the project which is experiential in nature. Stony Brook University is attached to Stony Brook University Hospital. So every year, I pick a unit, I meet with the manager, and they give us a clinical problem to solve. So for example, this fall the students and I will be working with the surgical ICU and the clinical topic is nurse wellbeing. So, as we know, we’re in this post-pandemic world and wellbeing has really moved to the forefront. Things like burnout, compassion fatigue is very prevalent in the healthcare environment and just globally as humans. I think we’re just a little tired of living in this fight or flight for so long. And now we’re trying to come back from this. So this fall semester, the students will be working in teams to find a solution for the surgical ICU for nurse wellbeing. So what we do is we search for articles together, and that’s how they get to their solution. We use a framework we use Melnyk’s seven steps for evidence-based practice. So in undergrad nursing, even though it’s called the nursing research course, the students are expected to utilize the research that has been done on a topic to make changes to their practice. Our expectation is not for them to actually conduct research. That’s a PhD level thing, but according to our essentials in baccalaureate nursing, that our accrediting body tells us what curriculum to teach to the students, the expectation is that they know how to read the research, how to critique it, how to appraise it, how to synthesize it, and how to use the research to develop solutions. So from there, they work in teams of eight throughout the semester, they develop their solution, they put it into a video project, the six-minute video project, and I choose the top two projects. Those top two projects then move on to the implementation phase. So then the unit will implement and evaluate the solution. And in addition to that, we put in for posters at conferences. For example, last year, we had two posters at ENRS. I was assigned the course of research, I was like, “Oh, boy, how am I going to make this interesting?” …because we know that research content can be a bit dry. So I ran the course for a couple years, and I knew that I had to do something with it. And that’s where I started moving towards this more experiential learning opportunity for the students. And so far, it’s been going really well.

Rebecca: So I heard you say something about teams of eight, and I almost maybe had a heart attack, [LAUGHTER] just thinking about how big that team is, and how to manage that. Can you talk a little bit about some of the structures you have in place to help a group that size, which is relatively large, be successful?

Tina: Sure. So teams of eight… that means I have 20 teams in total. And we’re all reviewing the same articles. So then I know the answers to all the questions. And basically, Google Drive is my answer. Every team has their own folder, within that folder are subfolders, I have them buddy up and be assigned to a certain number of articles. As a team, they have like individual and buddy responsibilities, which is clearly articulated in a contract that they review and fill out at the beginning of the semester. So they have individual responsibilities, buddy responsibilities, and they have team responsibilities. And every single class looks the same. So by the second class, they’re already into the mode. I don’t throw them any curveballs, every class structure is exactly the same, so they know what to expect. And they have appraisal forms to fill out. They have tables to fill out as a team to keep all of their literature organized. And the structure that I have in place seems to be working because there’s very little confusion now that I’ve kind of worked out all of the kinks. And I also always keep instructions projected just to make sure that they are apprised of the flow of class.

John: You mentioned Melnyk’s, seven steps of evidence-based practice. Could you give us a brief overview of that framework?

Tina: Absolutely. So there’s many evidence-based practice models out there. Stony Brook goes with Melnyk, and there are seven steps and actually I begin with step zero, step zero is igniting that spirit of inquiry. And that’s one of my main end goals of the course is for them to stay curious about how they can improve practice as a nurse for their patients. So that’s step zero. And then basically, what we do is we take the clinical problem, and then we frame it in the form of a question, a PICO question. And that helps us to find our articles. So once we find our articles, we go through the articles, we decide what we’re going to keep, what we won’t want to keep, then we start to critically appraise these articles, review them, read them, understand them, the students put that information into a literature review table, which is just the main elements of each article. After we’ve appraised all of the articles, the next step is to synthesize all of the articles. So what is the bigger picture? For that synthesis class students do complete synthesis tables. And when they create these synthesis tables, now the beginnings of their proposed solution begin to emerge. So then students put their solution together based on the synthesis table. And then the next step in this process is to implement the solution and then evaluate the solution. And of course, dissemination is always the last step.

John: You also mentioned that you use collaborative testing on exams. I’ve done this with a two-stage exam process where people take the exam individually first and then submit that but then take it again as a group. It also appears to have been and that’s been tremendously successful. It’s also appeared to have been really beneficial in terms of student learning, and it’s just so much more fun to watch the students work in groups on exams, than it is to go over the exam the next day with the whole class. That collaborative exam format has been so much much better than I ever expected it to be. Could you tell us a little bit about how you do collaborative testing on your exams?

Tina: I absolutely adore collaborative testing. If you have to assess students using exams, this is really maximizing the use of exams. So in my courses, students take collaborative exams in teams of three. And as we know, the research says that collaborative testing may decrease test-taking anxiety, the students have to take a large licensing exam at the end of the program. So it may help some of these students with that, like you said, immediate feedback on test performance, it really scales back the number of questions, I don’t even do exam reviews anymore, because the immediate feedback that they get, they’ve reconciled any confusion on the exam, that an exam review is no longer required. It increases student engagement and collaboration. I love how they, like you said, they debate, they discuss, that peer instruction. There are some people out there who can read a book and retain 100%. But generally speaking, you’ll have a better chance of retaining more information if you’re teaching someone else versus reading a book. Of course, that just varies learner to learner. So that could be really something to hone in on when it comes to collaborative testing. So, yes, the traditional way is to take the test individually, and then they take it again, in a team. And in our program, the clinical courses like medical surgical nursing, pediatrics, all of those, I would always recommend to do individual than collaborative because you really want to assess that individual on their performance and understanding of the concepts. And so I teach research, and I teach leadership and management, these are non-clinical courses, I skip the individual part, and take them right to a collaborative exam. So for example, for my research course, the students don’t know who they are paired up with, or in a team with, until about an hour before the exam. They get two articles, a quantitative article and a qualitative one. And then they have a set of questions to answer. Essentially, we’ve been preparing for this type of exam throughout the semester. So they end up doing really well. In my transitions to professional practice, where I’m teaching leadership and management, that is a traditional final exam, multiple choice, select all that apply, type of questions. And again, I actually do it on Zoom, they go into breakout rooms, they share their screen, and they take the exam, there’s a scribe who enters the answers. And also when it comes to accommodations, kind of as a side note, I’ve been able to set up strategies for individuals that do have accommodation so that they can maximize their experience as well.

John: When I first tried this, I was so excited about how the students were reacting with the collaborative exam that I took a short video clip while they were doing it and sent it to Rebecca. She was working with me in the teaching center at the time. It was just a remarkably positive experience.

Tina: Do you notice a difference like I would say an estimate of 10 points between the individual and the collaborative mean.

John: Generally, yeah. And the group one is virtually always higher than each individual score, except in one case in my class, where one student had a higher score than his group, and that’s because during the group discussions the student gave in to peer pressure within the group. I encouraged him to be more assertive when he’s confident about his answer. But that only happened with one student on one exam.

Tina: that’s pretty rare. I just love just watching them engage like that. So I’ll pop into like the breakout sessions, and they’re collaborating and negotiating and it’s just fantastic.

Rebecca: You mentioned earlier about your research class having a project coming up about well being. And I think that’s a topic that we’ve been talking a lot about in higher ed in a lot of situations. Can you talk a little bit more about that project and some of the research that’s going into it and some of the outcomes of it?

Tina: Absolutely. I mean, wellbeing is such a hot topic right now in probably every type of job you could think of. And it’s interesting wellbeing is kind of always been in the background. And I think the pandemic really shoved it into the forefront where it really should have been. That really needs to be, in my opinion, the top priority of any workplace because if your employees are well, it has a positive trickle down effect. So it has gotten to the point now where our accrediting body who tells us the essentials that we need to teach to our students, they have added a wellness component, and we’re adopting these new essentials in the next year. These are new essentials for us to follow. So it made it into the essentials, which is very telling. And now faculty are charged with teaching students, monitoring students, about their wellness and wellbeing. So this was pretty timely, because of the pandemic, the clinical topic that we’ve been doing for this EBP project has been things like compassion fatigue, burnout. And now this year, we’re doing wellbeing. Last fall, we worked with the cardiothoracic ICU. And the EBP project topic was compassion fatigue. And we wove in a lot of wellbeing into the solution, which is actually kicking off on July 1. So this year, instead of doing compassion fatigue, which has a bit of a negative connotation, let’s flip it to the positive. And like I said, we’re working with the surgical ICU, and we want to customize a wellness solution for that unit. So in the meantime, by proxy, I can teach the students about their own wellbeing and their own wellness. So I have a lot of content in there, so that they’re learning about this clinical topic to help develop a solution, but they’re also learning about it for themselves. And I do a few things with them, and definitely evolving this as we move along. And I’m lucky enough that I have the graduating class in the fall and the spring. So I move it through from the fall to the spring semester. So in addition and educating them on the different ways to promote your own wellness, we start each class with a mindfulness activity. I have a sound bowl that actually a student gifted to me, we do meditation, mindful breathing, every class is something different. This year, I’m inviting students to lead some of these sessions. So I want it to grow so that other students can participate and lead us and it’s literally three to five minutes at the beginning of every class, all lights down, devices off, phones flipped down, and we just take the time to be as present as possible. And I also help them keep an eye on their level of burnout. And I give them the professional quality of life survey at the start of every class. And halfway through, I’ll do a comparison of statistics between the different cohorts. Because I have the traditional cohort and I have the accelerated one, we look to see how our scores are doing over time, just to have that educational component to it. And then also the Insight Timer app, that’s an app that you don’t have, I would highly recommend that you download it. It has so many mindfulness type of activities that you can do. There’s a journal, you can track your progress. They have classes, and even the paid version, which is I think, maybe $60 for the year, they offer so many different bells and whistles, it’s really just a phenomenal app to use if you’re looking to promote your own wellness. So the other thing I wanted, I attended that CIT conferences, is I would love to use ChatGPT to develop a wellness assignment. So I’m still thinking about the inner machinations of how that would work. But hey, you know, if AI is here, might as well see if we can use it to promote wellbeing.

John: And it’s nice to have that focus of using ChatGPT positively because this is something that’s going to be part of students’ lives going forward, maybe not this specific tool, but AI tools are not going to disappear and using them for good would be a nice alternative for the concerns that many faculty have about the use of these tools. During your presentation at the CIT conference, you also mentioned using a variety of edtech tools. What are some of the tools that you use in your classes?

Tina: Sure. So I’ve trialed some apps here and there, I’ve used Plotagon. I used Go Animate for Schools, which is now VYOND, just for them to create case scenarios in their leadership and management class. And based on feedback, the one that they really liked is now a bit pricey. So I tried a free version of an app, and it really didn’t go well based on feedback. And that’s how it works in education. You try something out and you survey the students and if the experience over time is really not positive, you need to move on to something else. But things that really have stuck is I told you about Kahoot! and Articulate 360. How I communicate with the students. I use GroupMe. I prefer to communicate with them using that application over Brightspace or traditional email. They join via QR code and I have them all in one group chat and I can post quickly. They could send me direct messages, they could post questions in our group chat. And it just seems to really streamline communication because we’re all competing for their cerebral real estate, they have a lot going on, a lot of deadlines, so I find that this GroupMe app is really helpful. And I also try not to spam them with too many messages, thoug. It really seems to work. And then again, Google Drive, I can’t even begin Google Drive for everything, whether I want to survey them or whatever it is, Google Drive has it for us.

Rebecca: So speaking of Google, [LAUGHTER] you mentioned earlier using a Google form for a mid-semester evaluation. Can you talk a little bit more about that, and how you’ve used that to make adjustments in your class for the latter half of the semester?

Tina: Sure. So a Google Form is a pretty nice way to just give a quick survey to your students, I do that in the middle of the semester. And I have to tell you, that’s where I get my best data, because they are in the throes of it. And my response rate is typically over 90%, as compared to at the end, where they’re kind of just fizzling out, tired, maybe a bit over it, generally speaking. So I don’t get the response rate in the final that I do in the mid semester, when I analyze it, very short, a couple Likert questions: What do you like? What don’t you like? …and if there’s enough of a theme in the qualitative questions, or in the Likert scales, I’m able to make changes prior to them departing from me, instead of waiting for the next cohort to come in. For example, some things that came up was: “It can be a bit loud in the classroom.” So I’ve done something to control the volume in there, because it’s a very active classroom, or we feel like we’re sitting around too long during the TBL activities. So now I have a mechanism for them to let me know when they’re done with their activities, so that they’re not sitting around waiting. So those types of things. If they say, “let’s skip the final exam,” then that’s not anything that I can honor. But I’ve gotten some really good raw feedback that’s helped me evolve my classes. I’m just always so grateful for the student experience, because they inform me where this needs to go. Another way that I use a Google Form is with team-based learning. Michelson says that you should have the team members evaluate each other on their team performance. And typically, this is done at the end. But I like to do it in the middle of the semester, where they’re evaluating each other so that they have an opportunity to remediate, and then by the end, hopefully, their team’s performance scores have gone up. The challenge, though, with a Google Form is it’s very hard for me to share the feedback back to the students, it requires a lot of copying and pasting. And there’s a lot of room there for error, human error. So currently, I do bring in the students that are rated poorly just to give them some one-on-one guidance on how to improve their team performance. But in the meantime, to work around that I did trial a product called Kritik that offers that ability where the students will get their feedback back. But I reached out to our Center for Excellence in Learning and Teaching. And right now what we’re doing, we have a sandbox, and we’re working on trying to do a Kritik-like type of peer evaluation in Brightspace, using PeerMark. And we’re getting very close to ironing out some of the finer details. So I’m going to finally have an evaluation where every student can see their feedback from their team members based on their performance, so they know what they’re doing well and where they need to improve.

John: You teach both face to face and also online. Do you use many of the same techniques in your online classes that you use in your face-to-face classes? How do you modify your class for online delivery?

Tina: So I do everything the same, except that it’s in an asynchronous format. So students really have to be self disciplined in an asynchronous online type of environment. The online classes that I teach are post-licensure undergrads, so they have their two-year Registered Nurse license, and they’re looking to get their four-year degree. So some of the assignments, we tailor a little bit differently just because they have nursing experience, whereas my pre-licensure students do not. So maybe the assignments vary a little bit, but the structure is the same, using Articulate. I don’t use Kahoot! with them, only because I don’t have them in front of me, but they do have the quiz. And they have the TBL activities and things of that nature. So it’s the same, but it’s just in an asynchronous format.

Rebecca: I know that we mentioned in the intro that you do some research on some of your teaching practices. Can you tell us a little bit about some of that work?

Tina: Sure. So a colleague and myself got IRB approval, and we’re just starting to do some research on this evidence-based practice project that the students do in my class. And we’re just starting off with a cross-sectional study. We have a valid tool that’s been out in the literature that measures their perceived knowledge, skills and attitudes regarding evidence-based practice. So, I’m not building logistic regression models or anything yet, but starting off with a cross-sectional study to understand pre and post, the beginning of class and at the end of their research class, if there’s any impact or change in their knowledge, skills, and attitudes regarding evidence-based practice. So that’s where I’m starting. And I’d like to move on from there eventually.

John: And speaking of moving on, our last question is: what’s next?

Tina: So, I just would like to continue publishing and presenting. And continuing my research. Like I mentioned earlier, I’d like to introduce an AI tool for wellbeing, and Stony Brook just purchased several VR headsets. And because my courses include a lot of content about compassion, wellness, well being, I would love to develop a simulation about empathy. I think that would be a fantastic use of VR, apart from like, typical clinical scenarios. And that’s really my plan for now.

John: Well, thank you for joining us. And when you do have some results from your research, we’d love to have you come back and talk about it.

Tina: Thank you. Definitely. I really appreciate you inviting me. This is a wonderful opportunity for me. Thank you.

Rebecca: Yeah, thank you for letting us use your class as a little case study for folks to think about ways that they could change, improve, and reconsider their own classes. Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

300. Episode 300 Reflection

This is episode 300 of the Tea for Teaching podcast. Whether you are a new listener or have been with us for all 300 episodes, we are very grateful that you’ve joined us on our podcasting journey. In this episode, we celebrate this milestone by reflecting on what we’ve learned and how the podcast has evolved.

Show Notes

Transcript

John: This is episode 300 of the Tea for Teaching podcast. Whether you are a new listener or have been with us for all 300 episodes, we are very grateful that you’ve joined us on our podcasting journey. In this episode, we celebrate this milestone by reflecting on what we’ve learned and how the podcast has evolved.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Now that we’ve reached Episode 300, we thought we’d take the opportunity to reflect on some of the changes that we’ve seen in higher education since we launched the podcast in 2017.

John: Today’s teas are:…

Rebecca: I have Blue Sapphire.

John: I heard you just stocked up on a trip to your favorite tea store.

Rebecca: Yes, I was really excited to stock up on my favorite and I have a couple new ones too. So maybe in some upcoming episodes, we can try those out.

John: And I have Bing Cherry Black tea from Harry and David’s which is made by the Republic of Tea. When we first got started on the podcast, much of the focus was on specific teaching practices and techniques and interesting projects. Most of our guests were people that we knew or guests who were within our professional networks.

Rebecca: in the spring of 2020, as we know, [LAUGHTER] the focus shifted to the challenges associated with remote and online teaching, and the challenges facing remote learners and instructors.

John: As we became accustomed to pandemic teaching, we focused a bit more on faculty concerns as we transitioned into the transformed higher ed landscape. Historically, higher ed had been designed to serve the elites of society, and while higher ed gradually became more open and students have become much more diverse, many residual practices have worked against serving the students that we have. During the pandemic, faculty became much more aware of the inequities facing our students as well as faculty and staff.

Rebecca: Yeah, so one of the things that we’ve been talking about quite a bit is this more holistic focus on the needs of our students and faculty as humans, and really generating and creating a much more inclusive higher ed environment. How do you see that moving forward, John?

John: One of the things we’ve talked about is addressing the needs caused by the increased demands on time for faculty, staff, and students. As we developed new teaching techniques and tried to build more structure into our courses, it put much more demands on faculty in terms of redesigning their courses, in terms of paying more attention to the needs of students, and providing students with more feedback. And that has led to issues with burnout, which we’ve addressed in a number of podcasts.

Rebecca: And you’ve never experienced that, have you, John?

John: The day is not over yet. [LAUGHTER]

Rebecca: Well, how do we think about supporting faculty as we move forward with all these demands on their time in trying to prevent burnout? We’ve talked about this in a couple of episodes, but as we enjoy summer and have a little bit of more downtime for some of us… maybe not you, John… and move into the fall, what are some things that we need to be thinking about for our own classes to prevent burnout?

John: One of the things that we’re trying to be careful with when we recommend new teaching techniques to faculty at the teaching center here, is that they change approaches gradually, that small changes, incremental changes, are much easier to accommodate than the type of rapid changes that people had to do when they first moved into remote teaching. And so I think we have to be careful in making sure that we maintain a balance and we don’t burn out ourselves, because we’re not going to be very effective in supporting our students if we’re struggling to get through each day ourselves.

Rebecca: Yeah, we need to be present, just like we want our students to be able to be present and have the supports around them to be present in their learning. I think one thing that we’re also talking about in grad studies in our office is really this increased stress on faculty, and how do we support faculty, but also how do we support graduate student populations through things like accountability groups, or ways where there’s another human for accountability, but also for support, and not necessarily a mentor model, where there’s a power dynamic, but really a peer-to-peer approach to connect people together.

John: And we’re running two reading groups this fall to address some of these needs. One of the reading groups is on Sarah Rose Cavanagh’s new book, Mind over Monsters. And the other one is the second edition of Jim Lang’s Small Teaching. We had done that a few years ago, but we’ve had a lot of new faculty since then. And while we try to reach as many faculty as we can in our workshops, there’s a lot of faculty who are still teaching in pretty much the same way as faculty were teaching a century or so ago. And we’re hoping that by encouraging small modifications in teaching approaches, it might encourage more faculty to participate in introducing active learning activities and evidence-based teaching approaches.

Rebecca: It’s really easy to slip back into past practices when we’re tired [LAUGHTER] and overworked. And it’s not surprising that people have kind of slipped back into assignments and stuff that they’re really familiar with to reduce the cognitive load around new stuff and the many stressors around. So having that added support to help faculty re-engage with some of those ideas is, I think, a really great idea at this juncture. And I love that Mind over Monsters is one of the reading groups as well, because mental health is such an increasing concern, not just for students, but also for faculty and staff.

John: And we’re very much looking forward to both of these reading groups. Among the things we’ve talked about more frequently since the start of the pandemic are the challenges faced by underrepresented and contingent faculty.

Rebecca: I think when we’re introducing new techniques, and we’re thinking about supporting students around mental health, or we’re thinking about evidence-based practices and engaging in active learning, we need to remember that contingent faculty or underrepresented faculty have different barriers or different obstacles in implementing these things… or even more pushback from students and implementing some of these techniques. So we really need to be cognizant of supporting each other and realizing that we don’t all have the same kind of supports in place. I think some populations of faculty are just overly criticized. And when they try something new, it’s not accepted in the same way that a more dominant group’s adoption of those same techniques might be.

John: And that’s true both by students as well as by their faculty peers. And one of the things that’s come up in many of the podcasts we’ve discussed are the biases in both student and faculty evaluation of teaching.

Rebecca: Yeah. One of the things that I think is on the minds of our faculty too, is, as we’ve seen increased diversity of our students, we’ve seen diversity in levels of preparation. And I think those inequities have always been there. But again, maybe it’s more visible now than it had been in the past. How do we work through that in our classrooms, especially in these more introductory classes as students transition into college?

John: Well, I think those inequities have always been there, but they certainly grew with remote teaching, because our students face very unequal resources in their school districts and in their households. And when people are physically in the classroom, they’re at least exposed to the same infrastructure within their institutions. But when students were taking classes from home, as we talked about in many, many episodes, during a pandemic, they had very unequal network access, they had very unequal computing facilities, they may have been sharing a computer with multiple family members, they may have been forced to work. And as a result, the inequities in prior education and prior learning became much more dramatic during the period of remote teaching. And that disproportionately affected students from low-income households and low-income school districts. And what we have to do is provide resources, I think, for all students to be successful. And while we always should have been teaching, or providing resources and support, for all students, those needs have become much greater now, because while we are bringing in a much more diverse student body, we’re also losing students who come in with less preparation at some of the highest rates we’ve ever seen before. And we have to make sure that we’re providing the students that we accept with the support they need to be successful. And there’s lots of ways of doing it, you can build in some additional resources, you can connect to YouTube videos, and such things and provide support to students, you can use mastery learning quiz systems, and many other techniques. But we have to work towards having more faculty building that in because while many faculty are doing these types of things, and trying to build more support and more structure into their classes, it’s not a universal phenomenon.

Rebecca: And maybe even acknowledging that some students in the class are quite literally working harder to get to the same level.

John: The last few years when I’ve been teaching my large intro class, that’s something I’ve mentioned explicitly. I said, everyone here has all the resources they need to be successful. But if you had taken an AP introductory microeconomics course, or something close to that, you’re not going to have to work as hard to attain mastery of many of the concepts. If you have not been exposed to these things, or if your background in working with math and using graphs is not as strong, you’re going to have to work a bit harder. And that’s not a message that a lot of students appreciate hearing. But if we want to get all of our students to the same level at the end, the students who do come in with a weaker background need additional support to get there. And using tutoring when available, encouraging students to come in and talk to their professors and use office hours, all of those things can help but we’ve got a ways to go. What are some of the things you do to try to provide support for the increasingly diverse student body?

Rebecca: Yeah, I think the reality is that what we give each student isn’t the same, because what they come in with is not the same. So I often am trying to assess where students are and then pushing them right at that moment where they’re at, rather than expecting everyone to be at the same point. And I can do that a little more efficiently in a small class than you can in a bigger class. But I think we need to use those smaller classroom spaces to be able to do that so that everyone feels challenged, but also has what they need.

John: My introductory course is a prerequisite for all upper-level economics classes, and most of my students will be moving on to upper-level classes, so they have to reach at least a minimum level of proficiency in the discipline in order to be successful in their future classes. In other classes, instructors can be more flexible, and just try to get the most learning gains in their students, no matter what their starting points were. In my introductory class, at least, I have to pretty much take the students where they are, and try to get them all to the same place, while making sure that they’re all challenged. And that’s a very challenging goal to reach.

Rebecca: …and the difference between teaching those introductory classes versus higher level classes within a discipline, for sure, I think one of the most efficient things we can do is making sure that all students know the most effective ways to learn, because they don’t necessarily know those things coming into college, or even into graduate school. What we need to just remember is learning isn’t something that we just magically know how to do, we need to learn strategies and techniques that are effective.

John: We’ve had a number of guests over the last year or two who’ve talked about books that they’ve provided, or resources they’ve created to help students be more successful. Because one of the things that’s been pretty obvious for quite a while is that the study techniques and the learning strategies that students use are not generally consistent with what evidence suggests is most effective. And as a result, students are not using their time as efficiently as they can, by engaging in strategies that they perceive as being useful, that really result in very little increase in long-term recall… strategies such as highlighting, repeated rereading, and so forth. And one of the things that might be helpful is if we all could shift students a little bit in the direction of using evidence-based learning strategies, and some of that could take place through course design, by building infrastructures that incentivize the use of these techniques.

Rebecca: Yeah, and I think the moment that students realize that they don’t have the most efficient way, or the moment that a student begins to struggle is different, depending on some of that background, that they have. Students that come in well prepared may have never really struggled in high school, and maybe eventually, maybe even in the first year of college, you don’t struggle, but maybe it hits a little later on in their education, maybe not until graduate school. And then other students might struggle the second they get to college, because there’s not as much structure in place as there was in high school. So I think we need to be underscoring these techniques at all levels, and not just in their first year.

John: And one other thing that’s been discussed in many podcast episodes, is the importance of making the hidden curriculum of higher ed transparent to students, so that we don’t expect students to know what a syllabus is or how it could be used, that we shouldn’t expect students to know what is expected on a term paper in a class without making those expectations explicit and transparent to students. Because in general, we see a lot of students coming in, and they see it as a game where they’re trying to guess at what instructors are asking. And many of those guesses, especially for students who have not been in college prep classes before, are wrong. And they wasted a lot of time and effort that could have been spent more productively developing their understanding of the subject matter.

Rebecca: And the reality is that there’s differences between disciplines and between courses. And so the more we can be explicit about expectations within our own discipline, and within our own courses, and beyond the classroom experience of higher ed, because there’s expectations in other spaces as well, like student clubs, athletics, and all of the rest of the co-curricular activities that support student learning are incredibly important. And those are also not obvious.

John: One of the things that we’ve talked about much more on the podcast, and higher ed in general has been addressing much more extensively since the pandemic, is alternative grading approaches. Because traditional grading approaches and traditional course structures generally incentivize students to cram and to focus on maximizing their grades, rather than maximizing learning, so that if we really want students to shift to evidence based learning strategies, it would be really helpful if we could shift students emphasis away from grades and faculty emphasis away from high-stakes assessed activity and shift it more to activities that result in deeper learning, more long-term learning. And we’ve talked to many guests who have shifted to using strategy such as specifications grading, mastery learning systems, portfolio assessments, and ungraving, which has become one of the most talked about topics in higher ed in the last few years.

Rebecca: Yeah. And I think one of the things that comes up in a lot of those conversations is concerns over students just wanting the right answer and not learning and not critically thinking about the subject matter and the knowing of why and how, and doing analysis. And I think every instructor [LAUGHTER] has a desire for some of those kinds of conversations to come out in their classes, rather than just regurgitation of things that they’ve said in class.

John: Part of the issue is that when we get students in college, they’ve already had 13 years of experience in K through 12, where grades were the primary area of focus. And as a result, it’s hard to shift that focus from grades to learning. Besides alternative grading, we might use some other strategies such as encouraging students to be more reflective on their work, to spend some time in reflection-based activities and metacognitive development type activities.

Rebecca: Yeah, I know, this is a space where I was maybe a little hesitant at first thinking like, “Oh, these are just quick assignments that have no meaning,” but quickly realizing actually the value in really good well designed reflective activities that challenge students to think through how and why they learned something and what it is that they actually got out of an activity. And I’m often very surprised about how much learning occurs that is not visible, despite the fact that I teach studio classes, so I’m with my students much more than the average instructor. So I actually do observe a lot of learning. But in the reflection activities, I’m hearing a lot about how students are spending their time or things that they really struggled with and worked through that I wasn’t aware of. It also helps me understand where they’re not aware [LAUGHTER] of their own learning, or where they’re using strategies that aren’t as effective and helps with interventions. I know you’ve done a lot around metacognition, especially in your lower-level classes, but also in your upper classes.

John: In at least a couple of my courses. I’ve been using the metacognitive cafe discussion forum, which was actually the topic of our second podcast, Judie Littlejohn and I jointly developed this quite a few years back. And it’s been remarkably effective. It’s basically a low-stakes discussion forum that I’m using in my online classes, where students will reflect on their learning and share their learning strategies and will also read a bit about retrieval practice and spaced practice and the benefits of sleep in learning. And every time I do it, even though it’s only a trivial portion of the grade, it’s 5% of their total grade for participating in that activity, the students report that it was the most valuable learning experience they had in the class. A large proportion of the students at the end of the terms say they wish that they had learned these things back in elementary school, that they had been using practices that were not efficient and they didn’t realize that because they’ve never been taught how to learn. And it’s something that students have found really valuable. And the other nice thing about it is because, in this particular case, it’s done in a discussion forum, it helps them build community and helps them get to know each other, because they’ll often talk about the challenges they face. In online classes, many of the students have families where they’re taking care of young children, they may be working different shifts, they may be faced with other challenges that normally wouldn’t come up in a content discussion forum in an online class. But when they share that, and they share those challenges, and they share their career expectations, and they talk about how what they’re learning might be useful in their expected careers, besides the sense of connection, it also helps students see the relevance of what they’re doing and sharing that with other students helps build a little bit more intrinsic motivation in learning.

Rebecca: It also seems like there’s a bit of an immediacy in that context as well, because the information can immediately be put into action in a real lived experience and not something that may feel abstract, which sometimes happens within a discipline when it feels like maybe it’s not a thing I’m going to do anytime soon, professionally. So I think this really highlights the reason why we need to help students hook into everything that we’re doing to make it feel like they have a personal, professional, or educational connection to their own goals.

John: One of the topics that I use in each class where I’ve done this, at a point where students face the first really challenging material in the class, is just asking them to discuss how they deal with challenges. They share useful strategies, but one of the main benefits of that is it normalizes the sense of struggle, that when students are struggling with concepts, they often feel that they’re alone on this, but when they hear that other people are struggling with exactly the same issues and exactly the same concepts, it normalizes it, and again, it helps them understand that challenge is an important part of learning, which is not the message that they’ve generally received throughout their prior educational experience before coming to college.

Rebecca: It seems to me like this is the same reason why our reading groups work so well for faculty development as well is this connection among peers, but also that the challenges we experience are not in isolation. [LAUGHTER]

John: One of the things I’ve always enjoyed about the reading group is when people will come up with a technique or describe something they’ve done in class and people in completely different disciplines who might never have considered that will make connections and see how they could do something similar in their classes. That understanding that we’re all facing the same challenges makes it much easier to deal with some of the day-to-day stresses that we might have in our classrooms.

Rebecca: I know that one of the things that has come up in our reading groups, and also in our conversations about the future of higher ed and where we’re going to be going over the next few years is student engagement, and then specifically, the role of AI tools like ChatGPT [LAUGHTER] in the conversation. So if students don’t feel motivated, and they’re relying heavily on these tools, how do we get students to re-engage with the idea of learning?

John: Well, going back even just a little before the introduction of ChatGPT, which kind of hit higher ed by storm in late November of 2022, we did see a dramatic increase in the use of sites such as chegg.com, and various other sites out there, where the use of those tools became normalized a little bit, which made it much more challenging to give online assessments. And I think that’s where most people are concerned right now about things like ChatGPT, because with other places, you could at least locate where answers were coming from. And you could address that with the students and attribute it to the specific sites where they got their answers, which was, again, a bit of a challenge. But ChatGPT is raising some challenges for assessment that are going to be difficult to deal with, because it’s much more difficult to determine who is the author of specific items submitted online for assessment. And a lot of people are struggling with that right now. I know I’ve been struggling with it. In my spring 2023 online class, the quality of student writing on essays improved fairly dramatically over the course of the semester. And that seemed to correlate with the spread of the use of ChatGPT a bit. AI tools are really powerful, and they can be really useful. And they have a lot of potential value in education and in providing support for personal and work productivity. Right now, I think, more people are focused on the challenges, but we’re going to have to start thinking about ways in which we can productively integrate this and prepare students for a world in which the availability of AI tools will be ubiquitous.

Rebecca: And you teach in some really challenging contexts, really large classes in person, a number of online sections, and I know ChatGPT is keeping you up at night. What are some things that you’re thinking about… maybe haven’t resolved… but that you’re really thinking about redesigning or rethinking or retooling in the fall to just respond to the moment that we are currently living in.

John: As of 2020, I had shifted all the quizzing to online quizzes and tests and midterm exams and so forth. I’m seriously thinking about in my large face-to-face class, moving back to at least in in-class midterm and an in-class final exam. I really appreciated the fact that I could let students do it at their own pace, and that it took some of the anxiety and stress away when students did not have this two-hour time limit to complete an exam in the classroom. But with the size of the class, a large proportion of the testing is done with multiple choice exams, or algorithmic questions, and those are types of things that ChatGPT answers really, really well. Not too long ago, someone posted that ChatGPT 4 received a score of a 99th percentile on the Test of Understanding in College Economics (the microeconomics version of that), and those are the same types of questions that I’d be giving students on these quizzes. And while I had 1000s of questions that I had created that students were selecting from, all of those questions now are vulnerable to the use of AI tools, which makes it much more difficult to assess in that large class. Right now, the only thing I’ve really thought about doing differently in my large class is moving back to at least a couple of in-class exams. Now some of the things I was doing, such as polling questions embedded in the class activities and working on problems in class, where students submit that in real time, are generally much less subject to that type of issue. I know there are tools where students can scan the questions and so forth, they get responses back a bit more quickly, but it wouldn’t be as easy for them to do in real time when they’re in a polling environment. One of the main benefits of that is when I use polling, it was always tied with peer-to-peer discussions. And those peer-to-peer discussions is where most of the learning actually occurred from those in-class problem-solving exercises. For my online class, I’m not sure what I’m going to do. One thing I have done in the past is I’ve had students do podcast projects. And again, it’s pretty easy for chat GPT to generate scripts, but these projects are pretty heavily scaffolded. Students submit a proposal and they go through a number of steps to get there. And projects that are scaffolded like that, are probably a little bit less sensitive to the use of AI tools to generate the entire project. What are you thinking about in terms of your classes, or in terms of the graduate program?

Rebecca: I think we’ve talked a lot about the concern over the validity of our assessments and wanting there to be accuracy, not just for our sakes, but for students’ stakes in the value of their degrees moving forward. Part of it, I think, is really engaging in conversations around ethics around these tools, and not necessarily discouraging the use of the tools entirely, or banning the tools. I think that just motivates people to want to use them more, but rather to use them in ways that are productive, or interesting, but are also well documented… [LAUGHTER] like students are disclosing what they’re doing. And we can analyze the use of the tools in particular ways because maybe it could save time in particular places and not take away from certain kinds of learning, as long as we keep the learning objectives [LAUGHTER] up front. And then we assess when we’re using particular tools to determine whether or not it’s taking away from the learning. But I think these are hard conversations to have, and certainly not things that I want to be policing.

John: And I’d much rather not be policing these things. Sometimes students haven’t given me much choice in that. One example that I’ve seen recently is students submitting exam responses that asked him to analyze recent data, where the response said something to the effect: “as an AI tool, I do not have access to this data.” And when a student submits work like that, it’s pretty clear that they haven’t even read the essay responses they’re submitting on that graded assessment activity. And we want to make sure that students do actually interact and engage with their learning materials. Perhaps we can also design assessments that are not as vulnerable to AI-generated text. This semester, with my online classes, one thing I have shifted to, instead of having them discuss general debates or issues in economics, I have them focus on interpreting videos online, for example, where economists are debating certain topics, or doing readings that are not in the training database for ChatGPT, which means it’s much harder for AI tools to generate responses when they don’t have access to the underlying content that’s the focus of the assessment activity.

Rebecca: Would hyperlocal situations or examples also be a strategy because there’d be less widely available information on something like that.

John: Definitely. Information on the local community or the campus community or other local things, information that would just not be part of the training database is a good place in which we can ask students to connect the materials their learning to real-world events so that you maintain that sense of relevance while ensuring that the students are actively engaging with the work themselves rather than using a tool.

Rebecca: One of the things that I’ve used historically in my design classes, and it’s a little easier again, because I teach studio classes and see students more often so I tend to have a hunch as to what they’re working on, because I’m seeing them working on things, is really documenting process and not just using language, but showing through a video and showing steps along the way that might not be as easy to capture as an end product using an AI tool.

John: In general, open pedagogy projects, too, could be less vulnerable to having work being done entirely by AI tools. So videos would be an example, wikis perhaps might be.

Rebecca: I think that things that combine text and image are more challenging to have an AI tool create, at this moment.

John: That may very well change…

Rebecca: …by the time this episode comes out. [LAUGHTER] I think one of the things that I’m hearing us say actually, is that a lot of the strategies to reduce intellectual integrity or academic integrity issues around ChatGPT are also the things that are more likely to engage students and foster their learning anyways because they’re more authentic assessments, they’re probably more project based, they’re probably more long term with milestones along the way. And these are things that students often deeply engage in. And I think when they can connect to their local community, whether that’s the campus or the community that campus is situated in, or even their own hometown, in different ways around the discipline, those are all ways that students get a hunger to want to learn more.

John: And going back to our earlier discussion of the importance of shifting students’ focus from grades to learning, students are using tools like ChatGPT to raise their grades, even though they recognize it does not support their learning. If we can shift students’ focus to recognize the value of learning as improving skills that they’re going to need later in life, that should reduce the incentive for students to use shortcuts to avoid learning material.

Rebecca: If we’re not just looking for the right answer, but the journey to an answer, and even if it’s an incorrect answer, being able to understand why it’s not correct, and allowing that to be the learning is a really different way than our education has historically worked. The future of higher ed seems really stressful, John. [LAUGHTER]

John: It does, but it always has. That’s nothing new. But certainly the last few years have seen a lot of rapid change that… I hate to use the word unprecedented… but that have been relatively unprecedented.

Rebecca: And I think it really does speak to this need to connect with other colleagues, where we can share some of the challenges that we’re facing and brainstorm together to improve our teaching, but also to improve the level of stress we’re experiencing. [LAUGHTER]

John: One of the other things that we’ve talked about, especially within the last year or so is growing faculty concerns over student engagement. When students first came back to the classroom, there was a lot of excitement about being back. But since then, faculty generally seem to be noting that the level of engagement of students has shifted or has changed somewhat. More students are not completing assignments. Students in some classes have been disappearing from class as the semester progresses. And there’s a lot of concern that students are not as fully engaged with their coursework as they had been prior to the pandemic. So Rebecca, how are we going to solve this?

Rebecca: That’s a good question, John. I think one of the things that this aligns with is the higher incidences of loneliness, and mental health. And so finding ways to connect students to each other, and establishing those peer networks, I think, is one of the most important things that we can do in our classes. And it’s something that I’ve maybe always done in some way. But I’m being much more intentional about moving forward, because I’m feeling like even if students are in the same room, they’re still feeling really isolated. And so we have to be intentional about creating those opportunities for students to experience connection and feeling like they want to show up for each other and for themselves,

John: …using more group activities in class where the work of each student depends on the contribution of the other members does help create that sort of pressure on students to be there for their peers, to be there for the rest of their team. And that could be very useful.

Rebecca: Yeah. And I think the key to that, though, is not just assuming that students know how to interact with one another, or how to depend on one another in a team context, but really scaffolding those learning opportunities that really start with making connections and establishing relationships, because it’s the relationship that’s going to cause the pressure to show up for someone. And I think when we’re seeing high incidence of like ghosting, for example, it’s because the individuals don’t feel connected to the people that they’re ghosting.

John: And there have been a lot of studies done recently that show the importance of a sense of belonging in student persistence. So helping students form those connections is really important, because we have so many students who go to college, build up a huge volume of debt, and then disappear without getting the degree which does serve as a signal that they’ve actually accomplished something as a result of their education. And they end up with more financial struggles than they would have had had they not started. So we do want to help students form those connections for their own sakes, for their own future success. And one of the books we used in a past reading group was Relationship-Rich Education by Leo Lambert and Peter Felton. And that summarizes a lot of the research on the importance of building community and building connections, and also provides some really nice examples of ways in which institutions can transform to help facilitate those connections.

Rebecca: As instructors, we have a lot of power in that space to help students feel a sense of belonging. We can do really simple things to make someone feel seen and if they feel seen, they’re more likely to feel like they belong. So personalized messages, getting to know your students a little bit, being approachable, calling students by name, all of those things help students feel like they’re a part of a particular community. There’s so much to still learn and to come together around. And so I know that we’re looking forward to having many more guests and many more conversations to help work through many of the questions and concerns and things that we’ve raised today and have been raised by our colleagues. Now, John, we always wrap up by asking what’s next?

John: Well, what’s next for me is I’m heading down to North Carolina to teach at Duke again next week. And I’m looking forward to this. And this time, I’m going to try avoiding getting run over by a car. So I can actually teach my classes down there and spend some time away from the hospital this summer,

Rebecca: #life_goals. [LAUGHTER]

John: Small goals are sometimes more achievable. And Rebecca, what’s next for you?

Rebecca: This summer, I’m looking forward to doing some more work on our graduate student online orientation, which we put together as we transition to our new course management system in the fall and also working with some colleagues on an accessibility online module.

John: And we’re looking forward to talking to more of our wonderful guests. I’ve really enjoyed the experience of interacting with so many great people doing some really good research and doing such good work in higher ed.

Rebecca: We’re grateful for all of our guests and all of our listeners. So thanks for listening

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

296. ChatGPT Chat

Since its arrival in late November 2022, ChatGPT has been a popular topic of discussion in academic circles. In this episode, Betsy Barre joins us to discuss some of the ways in which generative AI tools such as ChatGPT can benefit faculty and students as well as some strategies that can be used to mitigate academic integrity concerns. Betsy is the Executive Director of the Center for Advancement of Teaching at Wake Forest University. In 2017 she won, with Justin Esarey, the Professional and Organizational Development Network in Higher Education’s Innovation Award for their Course Workload Estimator.

Show Notes

Transcript

John: Since its arrival in late November 2022, ChatGPT has been a popular topic of discussion in academic circles. In this episode, we discuss some of the ways in which generative AI tools such as ChatGPT can benefit faculty and students as well as some strategies that can be used to mitigate academic integrity concerns.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Betsy Barre. Betsy is the Executive Director of the Center for Advancement of Teaching at Wake Forest University. In 2017 she won, with Justin Esarey, the Professional and Organizational Development Network in Higher Education’s Innovation Award for their Course Workload Estimator. Welcome back, Betsy.

Betsy: Thanks. It’s so good to be back.

John: We’re really happy to talk to you again. Today’s teas are… Betsy, are you drinking tea?

Betsy: Yeah, actually, I was really excited. I’ve Chai spice tea. I was really excited when y’all invited me back because I’ve actually made a decision to stop drinking coffee as much as I have in the past. So I thought I’d be into all these exotic teas by the time that we recorded this, but nope, just a boring chai tea for today. But maybe next time when I come back, I’ll have some interesting teas for you.

Rebecca: We’ll make sure we ask you to level up next time, Betsy.

Betsy: Great.

Rebecca: I have a cup of cacao tea with cinnamon.

Betsy: Nice.

John: And I have a pineapple ginger green tea today.

Betsy: You all are inspiring me. I love it.

Rebecca: Did you say pineapple, John?

John: Pineapple.

Rebecca: Is this a new one?

John: No, it’s been in the CELT office for a while. It’s a new can of it, It’s a Republic of Tea tea.

Rebecca: I feel like it’s not one of your usual choices.

John: You said that the last time I had this. [LAUGHTER]

Rebecca: Yes, I just don’t associate this tea with you.

Betsy: You have a block.

John: I think I’ve only had it on the podcast two or three times.

Rebecca: Just a couple. [LAUGHTER] I just don’t remember. Clearly. Okay. [LAUGHTER] We’ll move on. We’ve invited you back, Betsy to talk about ChatGPT. We know you’ve been writing about it, you’ve been speaking about it, and everyone’s concerned about it. [LAUGHTER] But maybe we can start first by talking about ways that faculty might use tools such as this one to be productive in our work.

Betsy: When I discovered ChatGPT, the way that I discovered it, which was back in December, was that I had a colleague who sent a screenshot of asking them to draft a syllabus. And so my first encounter was actually with ChatGPT, doing something that would help teachers. It’s also the case that I’m a teaching center director, so, of course, I’m thinking of these things, but it has certainly shaped what was possible. And it blew my mind what it was capable of doing in a great degree of detail, actually. And then about a month later, I was working on a curriculum project where I was having to draft learning outcomes. And that’s a task that we do in the teaching center a lot, and always getting it precisely right and not really sure what’s the different ways that we can phrase this so that it’s actually measurable. And so I just started playing around with what was its capabilities in terms of learning outcomes, and I saw that it was actually pretty impressive and generative there. And then back then, when there was only GPT3, I kept trying to see if it could do curriculum maps for us. And I really had to force it and really think hard about my prompts to get it to actually map outcomes to courses and curriculum. But then when GPT4 came out, I tried it again. And I thought I was going to have to do it step by step. But this time, I tried with a philosophy curriculum, and I said, I want 15 courses, I want them to have three to five outcomes each, students need to take a certain number of courses, we want them to have each outcome three to five times and just sort of gave broad guidance. And it gave me a full curriculum as well as a map. And it was actually a very good philosophy curriculum. So it came up with the outcomes. It came up with the courses, I was floored, and it was my first request. So there are many other things I think we can use ChatGPT for in terms of our teaching, but the curriculum was really, I think, one of the most complex things that I’ve seen it do.

John: I saw you do that. And so I experimented to have it develop a whole major program, with course descriptions and learning outcomes for the program, as well as for each individual course. And it did a remarkably good job of it.

Betsy: Yeah, I was amazed because I didn’t really give it much of a prompt. And it had within the philosophy major, like comparative philosophy, issues of diversity, environmental philosophy. So it wasn’t the typical things that you would expect in a philosophy major, it was actually quite innovative in some ways. And I appreciated that. From the perspective of a teaching center consulting with administrators and faculty on curriculum, one of the things we often see is that the little blurbs in our handbooks or bulletins for students to see the descriptions of the courses, they’re about 150 words. And often they’re very much teacher centered. So here’s the topic of the course: in this course, you will study this, this, and this. And one of the biggest challenges is how do we turn those into outcomes. And so I actually tried to do that too, is I went through our bulletin and just threw in those 150-word descriptions of the topics, and had them develop three to five outcomes that were measurable. And it did pretty remarkably. And so I think that could be a useful starting place. Again, with a lot of this stuff, you don’t want to just take it as is, but a useful starting place to help our faculty and our curriculum committees brainstorm. And in about a week, we are going to do a course design institute at Wake Forest. We do it every summer, and I’m really eager to have my colleague Kristi Verbeke, and my other colleague, Anita McCauley, experiment with using ChatGPT as part of the process in the course design institute to see if it helps them speed up or get more ideas as they’re generating various aspects of design of their course, not just outcomes, but all the way down the line of the steps of course design.

Rebecca: Sometimes it can be really hard to get started, but as soon as you have a start, you know what you want.

Betsy: That’s right. And one response you might imagine to the fact that ChatGPT can draft learning outcomes is you might imagine someone saying, “Well, that’s a clear sign that it’s pretty easy and meaningless tasks to think to be able to draft the learning outcomes.” But what I have found is that when I, not just my colleagues, but when I have a really concrete learning outcome that’s measurable, it helps me design the course better, like it’s just so much easier to think immediately of an assignment. But when it’s vague, and it’s kind of like, I don’t really have it really clear, in my mind, it’s so much harder to do all the other steps. And so even if we think it’s a somewhat trivial task, having ChatGPT help our colleagues come up with really clear learning outcomes will help speed up everything else, at least that’s my hypothesis and we’re gonna see how that goes this summer.

Rebecca: We’ve played around a little bit of using like those course descriptions that might appear in a catalog and turning it into marketing language, which is very different.

Betsy: Oh, that’s so interesting. And has it worked well?

Rebecca: Yeah, I think it’s definitely a starting place to move it to a different kind of language.

Betsy: So I’m teaching a first-year seminar in the spring, and I’m an ethicist, so I teach a course on sexual ethics. And the last time I taught it, I had a pretty conservative title. And it was interesting, I only had women in the class, cis women in the class, there were no men that had signed up, which had not been the case before when I’ve taught that class. So I actually used it to say, I want to attract a diverse group of 18 year olds, or 20 year olds, first-year students to this class, what are some titles or some quick summaries that I might use, and it was really fun to see some of the ideas it gave me. I ended up mashing a bunch together, again, taking pieces of it as an expert and pulling it together. But it certainly got me thinking in a way that would have taken me much longer if I didn’t have that help.

John: I used ChatGPT to create an ad for the Tea for Teaching podcast just to see how it would work and I posted it on Facebook, and I got quite a few responses from people saying, “I use this all the time in my work.”

Betsy: Yeah.

John: This is a tool that’s out there and that came up really quickly, but it’s still a really early stage of this. And a lot of faculty are really concerned about issues of academic integrity, and so forth. And we can talk a little bit about those. But we have to prepare students for the world in which they’re living. And the world in which they’ll be living is one where AI tools are going to be ubiquitous. So you do a lot of work with ethics. How can we help students learn how to ethically use ChatGPT, in college and beyond?

Betsy: Yeah, I think it’s actually a fabulous question. And one of the things I’ve often said, a lot of folks come to me to talk about ChatGPT in terms of teaching and learning. And of course, I have lots of thoughts about that. But I actually have been particularly consumed with reading about the much bigger questions about what AI means for humanity, to be quite frank. There are really dramatic and important questions that we need to think about. And in fact, I think sometimes what I have seen is sometimes people will think that that’s just hype: “Oh, that AI might take over the world, or that it might have these dramatic effects.” But if you actually talk to people who are experts in artificial intelligence, they’re really worried. And when the experts are really worried, it makes me very worried. So when we think about preparing our students, on the one hand, you can think about it as preparing them to use a tool that they need to use for their career, kind of like, “I need to teach them how to use Excel, or I need to teach them how to do basic productivity tools.” And that’s really important. Don’t get me wrong. In fact, like a lot of students don’t learn how to use Excel, and they don’t learn how to use these productivity tools. I have colleagues that I’m teaching these things to where I’m like, “Oh, you didn’t realize you could use this, it makes your life a lot easier.” But I think the bigger issues are preparing them to think about the potential implications to really understand what the tool is doing and what that means for how we understand human intelligence, how we think about consciousness. I mean, what it means for whether we want to have a world in which there are artificial intelligences that we might have moral obligations to. I mean, all sorts of huge, huge questions. Now, I don’t think all teachers need to address those issues. Just like all teachers probably don’t need to teach the technical stuff. But I certainly think when we are thinking about curriculum, it’s essential that our institutions think about helping our students think critically and philosophically about what artificial intelligence means. And I think perhaps my guess is like some of our students or like, our faculty haven’t played around with it a lot or kind of like, “it’s just another thing like Grammarly, it’s not that big of a deal.” But we have found at Wake Forest that when we invite experts in, so linguists, or computer scientists, or machine learning folks, or ethicists to come and talk about these tools and how they really work. Folks have their eyes opened, and then realize, “Oh, this is a bigger deal than we thought it was and we might need to think about regulation [LAUGHTER] and what comes next.” So policy issues, not just ethics issues as well. So we don’t have an answer except for the fact that we need to be talking about it. I have some ideas myself about what I think regulation should be, et cetera. But I do think our students shouldn’t just be seeing it as a tool to make their lives easier, although it is, it also is important for them to think through the implications for society. And then I guess also as another ethical piece, obviously, is that, as we address the issue of academic honesty, helping our students think about their reasons for choosing to take liberties that they were not authorized to do and thinking about their own character. And that’s going to have to be an approach that is somewhat different than just punishment to help our students behave in ways that we wish them to.

Rebecca: I know that my colleagues and I have had some really interesting conversations around AI related to visual culture and creating visual items, because a lot of the libraries of images are copyright protected. And what does it mean when you’re taking something that has these legal protections and mash them up into something new? And then whose property is it? So they lead to really interesting conversations, and so you start thinking about it as a maker and your work being a part of like a library of something, and then also, when you’re using work that’s created, what does that mean? So one of the things that we’ve been talking about is there’s policy at all levels, like what’s our departmental policy around these things? And what kind of syllabus statements or things might we do to be consistent across courses?

Betsy: Yeah, and I think one of the most important things, and it’s gonna take some time, is for all of us to get clear on what we think our policies or our positions are going to be about what is appropriate and what’s not appropriate. And then once we do, to really communicate that to students, because I think they’re in a place right now where it’s all over the map. And many instructors aren’t actually sharing that with them. And so I think that gets us in a fuzzy situation where students assume “Well, if this professor said this, then this professor would be okay with it.” And often it’s very different. And so how do we at least have a conversation at the beginning of the semester with our students about what we think? And I actually think, as you point out, Rebecca, it’s a learning opportunity too for students to co-construct some of those positions. So let’s talk about the reasons why we might want to not just say it’s a free for all. We can talk about the value of art and the value of our work as artists, and what does it mean to just use somebody else’s work without acknowledging it? And maybe there are ways to acknowledge it. And unfortunately, one of the challenges of these image generators is that we don’t necessarily know what it’s drawing on. And so that’s one interesting regulation is: could there be a way? I mean, I don’t know. It’s tough. So one of the challenges with the science of this stuff is that often those who create it don’t know how it’s working. [LAUGHTER] And they will tell you that, that it’s a black box. And so to be able to get in there and say, “Well, I will reveal it to you.” I think sometimes folks assume they’re not telling us because they want it to be proprietary. But often, they’re not telling us because they don’t actually know how the algorithm was developed or is doing its work. And so that’s a really tricky situation. But when we did a number of series of workshops for our faculty this semester, and one of them was we brought in some experts, and we had some copyright experts and some lawyers that came in and talked about this, and really fascinating questions about copyright in our work that, again, is a great opportunity for students to learn that question in a real live way that they see happening.

John: Going back to the whole issue of copyright, in terms of human history, that whole concept is relatively new. And when artists created new work, they started by copying the work of others, and they added their own twist. And in general, in pretty much all academic disciplines, the work that people are doing now is built on the work that others have done before, this. Is what ChatGPT is doing, in part, just the same type of thing that humans were doing, except instead of spending years learning how to do this, and building on it slowly over centuries, it’s doing it in a few milliseconds.

Betsy: Yeah, and I’m not an expert on arts, and so I’m sure there are lots of experts, and Rebecca, you can jump in here as well. But I would say that there are certainly questions about: Is it harming? That’s the question, often with ethics, we’re asking. Is itt harming anyone to engage in this practice. And even if we don’t know we’re using somebody else’s work, we often are. Our ideas build on one another, etc. But of course, in a capitalist society where artists make money based on their work, there become new questions about how do I preserve my livelihood in this particular context? Now, again, if there was a different context in which we supported our artists, so that they didn’t need to make money off of their work, because we gave them a basic income, there may be a different question involved there. And so actually, I mean, I think the economic questions, so I’m tying you both together here. So economics and art, this is great. The economic questions are really interesting about what does this mean for the future of labor? And how do we think about work in the future? I mean, granted, now, it seems like it’s not going to be immediate, but there might be long-term implications for all of us that we need to rethink as well. So I don’t know, Rebecca, you have thoughts about that?

Rebecca: Yeah. I mean, I think it’s interesting. I mean, John’s pointing really to the printing press is when copyright came about, when there it was easier and less time consuming to make copies of things. And then in 1998 copyright law changed again, because of the ability of making digital files, copies, so easy.

Betsy: Napster. [LAUGHTER]

Rebecca: Yeah, copyright law hasn’t kept up with technology over time. So there’s constantly these conversations about technology and creative work and what is it mean? I come from computer art. So generative art is a thing that we do, and that’s algorithm-based and you would argue that the machine is collaborating, in some ways, you write the algorithm. So I think there’s a trajectory of this has been happening for a long time. But it does raise a lot of interesting questions. And I think it’s really important for our students to grapple with, and really critically think about, and for us to critically think about together. In some ways, it’s nice because it gives us something to have a good constructive conversation around and really sort through it together.

John: And then maybe a less positive note, in terms of the economics behind this, there have been a lot of stories of people taking two or three jobs on and using chatGPT, to do two or three times as much work as they did before. And one of the issues I’ve addressed with my students in my labor economics class is, if we have these tools that can do the work that college graduates used to do, will there still be a demand for college graduates to do these tasks. Most technological change in the past ended up replacing less skilled workers, and provided a really nice return to those who had college degrees. But this type of innovation might very well be hitting a little bit more heavily on college graduates than most previous innovations.

Betsy: Yeah, it’s hard to actually talk about this, because I feel like every week it gets better and better. And so I could say, “Well, currently, here’s the set of skills, if we’re an expert, we can use it to sort of level up a bit.” So as I shared with just the curriculum mapping, I’m able to ask it things, and then because I’m an expert, I’m able to do things with it and ask appropriate prompts that push it right in the direction I want it to go and then I produce this wonderful outcome, ehereas sometimes I tried just to show my faculty to put in questions from like physics or something and I couldn’t really assess whether it was a appropriate answer or not, or how I had to push it. And so there’s part of me that thinks that there will be roles for expertise. But then again, how good will it get? Who knows? Will it eventually out compete us? …which is somewhat of a worry… but I do think that, at least currently, there’s still a role for the expertise to play a role. But you’re right, it’s going to just make it more efficient so that we can do more. And then the question is, if we’re doing more will we need fewer workers? Or will we just be more productive? All sorts of interesting questions there. I will say just a funny little story about this point about computer art and economics is our office is called the Center for the Advancement of Teaching. And our acronym is CAT. [LAUGHTER] So we make lots of jokes about that. And we have a serious logo. But all along we’ve been thinking like, we should have a fun, funny cat logo, like with an actual cat, we just haven’t had the money for it. We have this wonderful designer we work with, she’s amazing, we got a quote, we’re going to do it if we have money left over at the end of the year, but I was just playing around with Midjourney and like, what can it do for me? …and I mean, it’s not as good as hers will be, I don’t think, but it was pretty remarkable, especially since this is just a fun logo, it’s not like our serious logo, that I could just use it instead of paying somebody to do it. So this is the real sort of challenge is that it’s really maybe just the most advanced things that we’re still going to rely on experts for but maybe some of the basic stuff that we would have paid, we’re no longer going to do and what does that mean, economically?

Rebecca: There are some existing sources prior to AI that were like people who didn’t have degrees or didn’t have a background in design, who would whip up something for five bucks. [LAUGHTER] And sometimes it looks like it was whipped up for five bucks.

Betsy: [LAUGHTER] You probably would think that about my Midjourney examples. I’m sure Rebecca, I’m sure. Yeah. It’s so funny. So, this is the other thing too, is the students will eventually get better at this, like googling prompts. I went to this website that was like, “Here are these professional designers who design logos, ask it to do it in that person’s style?” Or “Here’s some language that you can use for the prompt like: vector, flat,” like, well, this sort of thing, or a mascot logo, which I didn’t even know that was a thing. But I guess if I want a cat logo, it’s a mascot logo, learning those things, which I never would have prompted, it actually helped me get something that was a little bit better. But it is fascinating. Yeah. And I think that’s true, in my experience with the tool in general is that the more you use it, the more you learn what it’s capable of. And I do think that a lot of our faculty have not really spent a lot of time experimenting with it for a variety of reasons. They’re busy, et cetera. But I often encourage them to really spend as much time as possible with it to really understand what it’s capable of doing. I was sharing with John, before we started this podcast, that plugins are now possible with ChatGPT. And the plugins just take it to a whole next level beyond even GPT4. And I’m still starting to play around with that. And I think it’s just something that, again, faculty need to be prepared for, because right now they’re saying, “Oh, it can’t cite things, or it can’t search the web.” Well, now it can. And what do we do about that? How do we keep up with it if we aren’t paying attention to it?

Rebecca: I think one of the things that you said earlier and alluded to when you were talking about the logo is needing expert language and expert concepts to be able to curate the prompts. So if students want to use tools like this in a productive way, they then also have to have a certain level of expertise, presumably, to do a good job. If we want to encourage students to be productive and use a productive tool in a productive way, what can we do to coach students? Do we want to coach students in this way?

Betsy: The question about whether we want to coach students is a really interesting one. There are folks, I think, who are anxious that if you teach them how to use it, they’ll use it in inappropriate ways. And my sort of response to that is we’re gonna have to address their desire to violate norms in a different way. That’s a different issue. That’s an issue of character. It’s an issue of ethics, because I think they are likely to do it anyway. Now it’s true, if they don’t really know how to do it, we might find it easier to detect it. But I’m guessing that, in a year, it’ll be harder and harder to detect it, even if they don’t know how to do it. But as of right now, I would say I think it is useful to teach them. I wasn’t teaching the spring, but I am teaching in the fall. And I’m really excited to think about. I’m not going to totally redesign my course, some people have done that, I’m not going to do radical changes, but just to engage in the conversation with them in the ways that it can be used. And I think some of the most important, honestly, are using it to explain material that they didn’t understand or using it to interpret my prompts if they’re weird, or my expectations. So helping the students use it to help them with their learning. So giving me feedback on my work, and I have a whole list of things that I would recommend, which is somewhat different than what we immediately think about when we think about students using ChatGPT. We think about them using it to write their papers, or to start, brainstorm, or give an outline. And all of those things might be great. But I actually think, as a person who’s interested in pedagogy, and particularly in student learning, there are only so many hours in the day, I have so many students, I can’t be with them one on one for 15 hours a week. But if there’s a way in which they can have like a tutor, who’s there with them to say, here’s what I think might be the explanation of that thing you didn’t understand, or let me help you interpret this paragraph and put it in the words for a sixth grader, or I’ll give you an analogy related to sports, if that’s what you know. [LAUGHTER] All of those things are amazing opportunities for our students to accelerate their learning. And that’s what we want. So it is true that these tools can be a threat to learning if students are just using them to write their papers in a literally copy and paste kind of way. But I also think there’s real opportunity to help them accelerate their learning. And again, you have to be careful, because it’s not perfect. And that’s your point about expertise. But I think, frankly, sometimes the advice they get from their friends, or if they go to the internet and Google it or YouTube, they’re not getting great advice either. So it doesn’t have to be perfect to be better than what they’re currently doing, is I guess what I would say? So I think that’s important. And then we could talk if you wanted to about how they might use it as a writing tool. I think it’s trickier there, of how we could ultimately and I’m sure you’ve heard this before in the previous ChatGPT podcast you did, it ultimately depends on your learning goals. What are your goals for your course, how you want to use it, but I do think there are certainly legitimate ways in which we can help students use it to help them learn more.

John: And just following up a little bit on that. I’ve heard of a number of faculty who are encouraging students to use it to create tutorials on specific topics where they may have a weaker background. And that’s certainly a very good potential use of this.

Betsy: And I even have experimented with like, “Okay, so give me feedback on this and then give me a learning plan,” like, “give me an improvement plan,” like, “what should I do? What steps should I take to get better at this skill?” …and it’ll actually give you pretty good plans. And it also can help them with time management, you know, “I have this many things I need to do help me prioritize what I should work on next.” And that’s good for us, [LAUGHTER], but it’s also really good for our students who really struggle with time management, I think. So I really do think there are a number of things that students can use it for, that I would feel comfortable with, but I also think it’s a really useful exercise for everyone listening or for any instructor to think through the possibilities. And you may decide these things are okay, these things aren’t okay, and it may differ for each class. But that is really important to do before you can actually communicate to your students what is and is not okay. And if you want them to actually do what you ask them to do, you have to have good reasons, I think. So you can’t just give them a rule, you should justify that rule, like with kids a little bit, you got to say why you think it’s the case to hopefully bring them along of why they wouldn’t want to just use the tool straightforwardly.

John: One thing I’ve used with some students, especially when I’ve talked to him about some of their uses of ChatGPT is, if all they’re learning in the course is how to type a prompt into ChatGPT and copy and paste that in, what types of skills are they acquiring there that’s going to be useful when they leave, because they could be replaced by anyone typing in those prompts.

Betsy: Right? What makes them unique. So one frame for this is the things we’re teaching students in school are useful for them. We want them to learn so that they can be productive in the market. That’s one way we often frame the work that we’re doing. But I think this gives us an opportunity to open it up a little bit wider where we think about the purposes of education beyond just what is going to be useful in the market. And so I sometimes will use the example of pottery… we’re coming back to art… is that I took a pottery class, after COVID, but it was like the first thing I wanted to do after we were back in person and so I took a wheel throwing class, and I was absolutely terrible. But the idea that I would go to Target and just cheat by bringing in something from Target that was made by a machine. No, there’s a reason I’m doing it. I want to actually learn the craft and the craft has meaning in and of itself, apart from the fact that yes, I could get a much better bowl [LAUGHTER] from Target than anything I will be able to create, but I’m really glad I’m doing it. And I think that’s really what’s gonna start happening is we’re going to start to see that there’s actually intrinsic value to some of these tasks, apart from their value for the, you know, am I gonna make more money later, etc, that we actually think that learning and thinking and the creativity that comes with producing is a value in itself. And that’s going to take a while to turn our students in that direction again, because there’s so market driven right now. But if things start changing in the market, and there are fewer and fewer jobs, they may be open to that conversation.

John: And maybe with the growth of alternative grading systems that try to shift the focus away from extrinsic rewards to intrinsic rewards, this could be quite complimentary.

Betsy: Yeah, and I know that we don’t want to talk all about academic honesty, but it is a real question. And so I don’t want to dismiss the faculty who are anxious about this. I was sharing before the podcast was recorded as well that the news about Chegg losing so much money in the past few months, was a real indicator to me that perhaps my optimism [LAUGHTER] about students not using it was ill placed, that in fact, that’s pretty good indirect evidence that a lot of students are now using ChatGPT to do what Chegg used to do for them. And so it’s not good for us, but it’s not good for the students. And so we do need to think about it. But I do think there’s sort of two broad approaches. One is like the punishment and enforcement approach, and then the other is prevention. And I think focusing on prevention is really where we need to go. And so referencing focusing on the intrinsic value of the work, maybe pulling away from those high-stakes graded assessments is a way to think about motivational changes of how we prevent students from engaging in these and I sometimes will use the example again, of the pottery class, like the idea that I would be motivated to cheat in a pottery class is absurd, like, why would I cheat at that class, because I’m just doing it for my sake. Now, if I was doing it so that I could get more money or so I could get this grade so that I could get into something else I wanted, then I might be willing and tempted to cheat in the pottery class. We know that students cheat because of the grade, they don’t cheat just because they think that’s the fastest way to learn. [LAUGHTER] They know they’re not learning, but they’re like, “I need this grade, because I need this degree so that I can get this job.” And so really bringing them back, decreasing that external stuff, and taking them back to the value of learning may be the only way we’re really going to tackle this. Now, it’s easier said than done. We’re all in a system where grades matter, and students need to get degrees and so it’s a longer conversation. But I do think revisiting some of the literature on cheating, even before ChatGPT existed is going to be really valuable for all of us.

Rebecca: So you’ve talked about moving towards more low-stakes opportunities, and we’ve hinted towards alternative grading, what are some strategies that faculty can use to assess student learning? We’re concerned that we’re not able to see whether or not a student is learning if they’re using tools like that, those are the conversations that we’re having.

Betsy: Yeah, so there’s things you can do to hopefully try to prevent it. But those may not always work. So I have a lot of ideas for how to prevent it. You can give them extrinsic reasons for using the tool itself. Like for example, I mean, this is just a simple one, but let’s say you’re teaching a math class, and you have an in-person final, and you tell them, “you’re going to be preparing yourself for the in person final by doing the homework yourself.” So there’s a kind of extrinsic reward of the final that’s in person will hopefully motivate the students to do the practice problems themselves, because they need to actually learn the thing that will get them that reward. But I do think if they do it, so again, lots of motivational things to talk about. But if they do it, first of all, how do we know is a really interesting concern. And I think that one interesting point that I’ve raised when I’ve had some conversations with folks is that I think a lot of people think when we’re talking about we need accurate assessments of student learning. The first assumption is that what we’re talking about there is we need to have grades that are just, so when we pass them on to like jobs, or to future courses, that we have just grades. But I actually think there’s a real learning reason why we want accurate assessments, is that if I can’t accurately assess your skills, you’re not going to learn. I actually want to know where you are really struggling, so then I can adapt my teaching to better help you learn. And if it looks like you’re doing great, I’m moving on, I’m moving on, I’m not going to actually help you learn that thing. And so it’s really important for learning as well that we have really accurate assessments of their skills. And so if they are using it, so how do we detect it? Tough one, but I think that’s where multiple measures. So you might imagine you have some in-class things that are happening, you’re not just lecturing. So this is a good reason for active learning as well. Because you’re engaging your students in class, you actually hear them speak in class and explain things to you in class. And if they’re struggling there, and then all of a sudden, they have this beautifully written paper, I think that’s a useful comparison. It’s no guarantee that that’s the case because sometimes students need time to reflect, particularly English as a second language learners need time to build their arguments, etc, rather than just being on the fly in class, but it is interesting evidence and that there’s people talking about oral exams and other possibilities or at least having conferences with the students about their work. So it’s not an exam, but just like let’s meet to chat about this. Now, of course, if you’re teaching a huge class, that’s not possible and available to you, but those that are teaching smaller classes, it might be. So I think we’re gonna have to be creative. I have not found a silver bullet here, I have heard lots of great ideas of things that could be possible, but all of them have trade offs, all of them come with downsides. And this is kind of my mantra all the time, when I think about pedagogy issues is that we should not get too absolutist about this, all of us are going to make different choices. And they’re all gonna have different downsides, and they’re all pretty reasonable, because right now, there is no obvious solution of what we all should be doing. I think some may choose to do oral exams, some may choose to do in-person, others may choose to say, “I’m not gonna pay as much attention as some others are.” And all of those things I think are reasonable. They’re just different approaches. And we should keep paying attention and be open to changing our minds if it seems like it’s not working. But I don’t feel like it helps us to be in sort of like one strong camp or the other when we think about the issues of academic honesty, and ChatGPT. So again, I don’t have an answer. Just lots of questions for you. But did you find anything that was useful over the past semester for you?

Rebecca: We teach really different things so our approaches are going to be very different. In my classes, we’re doing creative work. And so historically, and we continue to do this, documenting your process is part of the project. And so we see a project evolve over time. And that maybe involves using the use of AI as part of an input during that time, but documenting that as something that makes that happen. And we do critiques, we show things in progress, and we talk about it, and there’s feedback that’s recorded at those moments. And then if we’re not responding to feedback, then we’re not growing. So we have some systematic ways of demonstrating some creative process there and having to discuss and determine decision making around design decisions or creative decisions, like “Why did you make that decision?” And if it’s just like a random choice, then let’s be intentional about it. And now you need to maybe rethink that choice and make it more intentional. So those kinds of authentic learning opportunities really do kind of push it in a direction where it’s a lot more difficult to use AI as the entire thing. [LAUGHTER] It might be a part of the process, but it wouldn’t be the final output.

Betsy: John, I want to let you respond too, but what you’ve done is that one of the things about that, because that is certainly like doing authentic learning and process-based stuff. As you put it, Rebecca, it’s more difficult. But that doesn’t mean, and this is important, some people will say is that like, ChatGPT will tell you it’s process too, or you can ask it to give me processes, etc. So I do think, actually, one of the things that I think is I appreciate about your example is there’s a lot going on in class, there’s a lot going on, and it’s harder for folks who are doing asynchronous online courses. But if there are ways in which we actually see the process, and that’s kind of the authentic too, is that we’re actually not assessing a product, we’re literally live with them watching the process, I think we might be more likely to get some accurate things. And then if we just said, “Okay, we want you to write about your process, we actually want to see the process as most important.” So John, what about you,

John: The classes I’m most worried about are my large class, which has up to 400 students in it, and an online class that’s on the same topic with generally 40 to 50 students in it. And there’s some challenges there. In the large class, one of the things I’ve done since the start of the pandemic, is to shift all the assessment to online activities. I used to have a midterm and a final that were cumulative, they weren’t a tremendously large portion of the grade, there were lots of low-stakes tests that they could do over and over again. But the validity of those I suspect is going to be a bit different now. Because ChatGPT can do quite well with multiple choice questions and short answer questions and even algorithmic questions. So I’m probably going to bring back at least a midterm and final in person in my large class, just for the reason you described, the motivational thing… that you can practice these things as much as you want to learn it, but you’re going to be tested on this. And the greater your ability to recall and apply these concepts, the better you’ll be able to do on these things. And I wish I didn’t have to do that, because there’s so much advantage of letting students do things over and over again until they master things. But I’ve looked at some of the times on some of the quizzes I used this time, and students were turning them in [LAUGHTER] much more quickly than would have been possible had they not been relying on some sort of assistance.

Rebecca: Well, John, they’re just learning it so much better.

Betsy: Yes, that’s right. That’s right. That’s right.

John: And a nice side effect is you no longer get any spelling or grammatical errors.

Betsy: Yeah, you can read it faster as well.

John: Yeah, it makes it easier. [LAUGHTER]

Betsy: Yeah, no, and I do think as much as I think that we should trust our students, and I don’t want to be overly alarmist. There’s a lot of evidence that our students are doing it and even our students who would prefer not to do it, I think are doing it because they perceive that all the other students are doing it. So this was the same problem in the pandemic with academic honesty is that you have some students who will never cheat for whatever reason, [LAUGHTER] a small number… you have some students who will always cheat, they’ll find ways, they’ll pay somebody, whatever… they’re gonna find ways, and then there’s just a whole bunch of students in the middle who If the context really matters, and if they assume that all the other students are doing it, it puts them at a disadvantage to not do it. And we shouldn’t put our heads in the sand or assume that all of our students are not doing it. We shouldn’t also assume that our students are these horrible people, because they’re doing it, we need to recognize they’re doing it, and how can we help them create the conditions where they would be motivated to not do it to get themselves in trouble. And I do think your point, John, about the 400 students, about teaching an async online course, and even Rebecca, some of your description of what you’re doing in class. One thing that occurs to me, and I don’t have any illusions that this is going to happen, but I do think what these push against is our traditional model of how higher education happens. So we assume for the longest time that it was a lecture that took place. So that’s why 400 didn’t matter versus 20, we also assume that most of the learning would take place outside of class, because you would just come to a lecture and then you would go read the book and learn and teach yourself, basically. It’s kind of this old school model of like, the professor is just there to give you information, you’re going to teach yourself before the exams. And I think I can imagine a world in which, if we really want to see process, we need to be with our students more than three hours a week, and we need fewer students in the course. But that would be such a radical change to the economic model of higher education. I can’t imagine how expensive that would be. But it is more similar to K through 12. And in some ways, I think K through 12 folks have more of an advantage because they’re with the students so much more that they can actually watch them. And homework is less important. One of the most important things I always tell my students and have for years is that most of your learning will take place outside of class, and to emphasize that to them. And I think maybe now that creates a challenge because we’re not with them. And so we can’t sort of see whether they’re doing what we want them to do. So we really have to lean into the intrinsic motivation pieces of what is it that motivates them to want to do well, but with 400 students, they don’t know you really well, John, so they don’t feel guilty about like I have this relationship with my professor. It is tough. And I guess I would say on this point about academic honesty, and maybe we don’t have to keep talking about academic honesty. But I’ve seen a lot of faculty feel really guilty about their approach to this on both sides, like either they’re too overly harsh, or they have ignored it too much. And they’re super anxious about whether they’ve taken the right approach to academic honesty. And I think the most important thing I would say to instructors is this is really hard. Don’t beat yourself up about it, like you’re trying your best. And none of us have a perfect system. If we did, we’d be able to sell that, and it would be great. [LAUGHTER] We don’t have a perfect system. Some of us are maybe leaning one direction, and others are leaning in the other direction. And it’s really demoralizing when our students cheat, and then that makes us depressed as well. But also know that you’re not the only one that all of us have students who cheat and that’s unfortunately, part of the educational process. So do your best. [LAUGHTER] Pay attention. But don’t worry if it’s not a perfect outcome.

John: One of the things I was struggling with just recently as I was grading exams is how do I evaluate the work which is clearly the student’s own work, versus one that probably wasn’t the student’s own work. I don’t want to penalize students for actually trying.

Betsy: I think some people say like, “Ah, let’s just ignore it. It’s not my job to be a cop.” But I think the reason we want to actually do that is an ethical reason, which is that I don’t want the students who actually put forth the effort to be disadvantaged. So I think that’s the right impulse, John, yeah, for sure.

John: One thing I hope that doesn’t happen, though, is that we move to proctored exams online, and that we don’t move to more use of high-stakes in-person exams and so forth, because that would go against so many other things that we’ve been arguing in terms of equity and inclusion and so forth.

Betsy: Yeah, and also, these detectors. So TurnItIn, which most folks are using now, because many schools have TurnItIn, attached to their LMS. And so even before schools have had an opportunity to make a choice it’s default turned on. So your institution has to choose collectively to turn off the AI detector in TurnItIn. And so I think that’s important, too, to think about, like, are we just going to move to these detectors as a way of punishing students? And are they reliable enough? We don’t know. So there are all sorts of good equity questions. Actually, there’s a paper that I read in preprints, about how the detectors seem to flag international students more than those who speak English as a native language, in part because their grammar is better. [LAUGHTER] And so it’s more formulaic, because we’re teaching them the formula of how to speak English. And we need to be mindful of like, how do we balance these things,our equity concerns… and really they both are equity concerns, as you point out, John… so there are equity concerns about more high-stakes testing and in-person testing, etc. But if we just ignore it, there are also equity concerns for the students who do the work versus the probably the privileged kids who are just going to be like, “Whatever, I’m going to pay my $20 a month for GPT4, and be able to get the better answer to be able to use it.” So how do we come up with some sort of solution that balances those and we probably won’t be able to have… at least I don’t think there’s one… where there aren’t some harms? And so it’s really about like, which harms are we willing to tolerate while we work for a better solution? And that’s the hard part of ethical reasoning is that there’s not a solution where no one is harmed usually in these dilemmas.

Rebecca: One interesting thing you said like spending more time with your students, which I have the luxury of doing in a studio art space, we spend twice as much time with students for the same credit hours…

BETSY. Interesting.

Rebecca: …which is valuable. We see process, we get to know our students really well, it’s a relational space [LAUGHTER] for forming relationships. And that really does change the dynamic. But that’s a really big time investment, while the cost of faculty in the spaces but also students of certain backgrounds, or if they have to work, it becomes much more difficult for them to take those kinds of classes, because they’re offered at particular times, and they’re longer, they’re harder to schedule their job around, and that kind of thing. So there’s equity issues in that space, too, as you’ve alluded to, about being in person.

Betsy: Yes, being in person, and then also the point about extra time. I was on talking about workload before. One interesting thing related to workload is that we know, from the research on student learning, that time on task increases learning. And so sometimes I think when we talk about making things accessible to students who are working 40, 50 hours a week, what we’re really doing is reducing the work that’s required of them, which is fine, if it’s just about getting the degree, which I think you can make interesting ethical policy arguments that that’s really important, because economically, it allows them to advance, etc. But if it’s about learning, we actually shouldn’t be reducing the amount of time they’re spending because they’re going to learn less. And so then there’s that tricky question of if we need students to spend 40 hours a week on school, what do we do? We have to compensate them so that they’re not having to work, there’s much larger policy issues at stake here beyond just like, well, we just got to expect them to buckle up and do it, they got to work 80 hours a week now instead. So these are all tough things. And in the context that we are in, where we don’t have those amazing, policy based governmental solutions in the United States, we have to make compromises. And we may say, “Well, maybe a little less work for the students who are working is the compromise we’re going to make for the greater good in this situation.” But recognizing it is true that they’re probably learning less if they’re not putting in the 40 hours. So but maybe with ChatGPT, we can speed it up, I don’t know. [LAUGHTER] So, interesting things about efficiency.

Rebecca: So, we always wrap up by asking “What’s next?”

Betsy: So I’ll speak to related to artificial intelligence a little bit. So I am, in July, going to the Council of Graduate School summer workshop, New Dean’s Institute. They’ve invited me out to share a little bit about ChatGPT, and I’m really excited to be thinking about what’s distinct about graduate education with respect to these tools, it kind of merges my interest in faculty use, as well as thinking about student use as well, for learning. In addition to that, we’ve been talking a lot about how to prepare for the fall, when the faculty come back. We were sort of… just like, COVID… sort of flying by the seat of our pants in the spring, like, here’s some things we’re gonna roll out for you, we’d like to be a little bit more intentional in the fall. And so as I’ve alluded to in this session, I really do think focusing on motivation for students is going to be really important instead of detection. And so we’re gonna do a reading group, we’re gonna go back to Jim Lang’s Cheating Lessons, which still holds up pretty well, actually. And we’re going to do a reading group of faculty on that. And then we’re also going to read the Grading for Growth book that’s just coming out in July, which we’re super excited about alternative grading. I’m teaching in the fall, as I said, so excited to actually try some of these things out and see if my ideas are actually practical [LAUGHTER] or not. And hopefully, I guess I just say, what’s next? I hope there’s some regulation. So we didn’t get into a lot of details about this, because we were focusing on teaching and learning. But I know Sam Altman, and Gary Marcus were before Congress. And I do hope that we actually see, unlike with social media, that we see some movement for some regulation about the development of these tools. So I think what we have now… fine, let’s figure out how to use them. But it’s really anxiety inducing to me that these tools will develop skills that nobody planned emergently like, it’ll just, “oh, now it has this new skill.” And the more that we build these tools out, we don’t actually know what we’re going to create. And I think [LAUGHTER] that is a little worrisome to me. And so I hope that what is next is more regulation on the tools.

John: We should note that we are recording this several weeks before it’s actually released. And we hope that at the time when this is released, [LAUGHTER] we haven’t reached that AI apocalypse that so many people have been worried about.

Betsy: That’s right. That’s good, John, thank you.

Rebecca: Well, thank you so much for joining us, Betsy. We always enjoy talking to you.

Betsy: Thanks for having me.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]