168. Synchronous Online Learning

The pandemic forced many faculty to experiment in different modalities in 2020. In this episode, we reflect on our own teaching experiences with synchronous online courses this year.

Show Notes

Transcript

John: As we approach the end of a really challenging year, we’d like to thank all of our guests who provided so much help and support to us and all of our listeners, and we’d like to thank you, our listeners, for hanging in there with us. We’ve all learned a lot in 2020 and we’re looking forward to a chance to apply what we’ve learned in circumstances in which there are fewer external threats.

…and now we return to our regularly scheduled podcast.

The pandemic forced many faculty to experiment in different modalities in 2020. In this episode, we reflect on our own teaching experiences with synchronous online courses this year.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Over the past few months, we’ve talked a lot about the pandemic and ways to adjust our teaching. And we’ve talked a lot about online learning, but we haven’t really focused on synchronous learning. John and I both taught synchronously this semester. So we decided that in this episode, we would focus a little bit more on synchronous learning and what we’ve learned about it in our own experiments in our classes.

John: Our teas today are:

Rebecca: I have Scottish breakfast once again.

John: …and I have a blend of spearmint and peppermint tea.

Rebecca: That sounds much healthier than my choice.

John: It’s not my first tea of the day.

Rebecca: This is not mine, either.

John: This is my first herbal tea of the day.

Rebecca: This is my second pot of the day. [LAUGHTER]

John: So, Rebecca, what classes were you teaching this fall?

Rebecca: I was teaching two design classes that are smaller. So I had one web design course that was stacked. So, it had beginning, intermediate and advanced students in it, 25 students, and we met synchronously, but also had asynchronous classes, it’s considered a studio course. So for a three credit course, we have six hours of class time, and three hours of outside work, which is a different balance than maybe other folks. And then the other class I was teaching was a special topics design course, which was smaller, it was about 10 students. And that class was also synchronous, but it was a project-based class, and we worked on two community design projects: one for a project called Vote Oswego, which was a get-out-the-vote initiative on campus, and the second is a project called “Recollection,” which is a storytelling project with adult care facilities.

John: And we do have an earlier podcast on an earlier iteration of Vote Oswego. So we’ll include a link to that in the show notes.

Rebecca: So John, what kind of classes are you teaching? We obviously don’t teach the same thing.

John: I was teaching two classes this fall. One was a large synchronous session with 288 students. And the other was a fully asynchronous section with 60 students this semester,

Rebecca: At what level were the students in both of your classes?

John: These were both introductory economics classes. So most students in the class were freshmen, and it was their first economics course.

Rebecca: So, your classes are much larger than mine, you’re teaching much more younger students or newer students, and my classes are smaller, project-based, and usually junior or senior students.

John: Yes. And there’s certainly some differences in the disciplines as well.

Rebecca: No, that they’re the same. [LAUGHTER]

John: Why did you choose a synchronous mode of delivery rather than an asynchronous mode, or a face-to-face option this fall?

Rebecca: So I chose not to do face-to-face delivery for my own health reasons, I chose to not be on campus for my own safety, because I have a chronic illness. So I chose specifically to have strong synchronous components, because a lot of our students are used to working in a studio together and having a community around each other and kind of feed off of each other’s work and work collaboratively. And I wanted, because of the classes I was teaching, to continue to have collaboration as a key part of my class. And I was really concerned that if I didn’t have a strong synchronous component, my students wouldn’t be able to effectively collaborate with each other, because there would be too much scheduling issues and what have you. So it’s a little bit of a carryover from the way that I would run my classes in person in that I give a lot of class time to project-based learning and team-based work and do a lot of lectures and things like that asynchronously in like a flipped classroom style. How about you, John?

John: Basically, I tried to preserve something as close as possible to what was originally scheduled or what was originally planned. And my large class is typically about 400 to 420 students, and I just couldn’t imagine taking that class and doing it in a completely asynchronous manner, because when I teach a class asynchronously, I give students lots of individual feedback, and it would be really challenging providing individual feedback to several hundred students. I just didn’t really have the time to do it in that sort of mode. So I thought it was better to work in a mode where I could give students feedback in a group setting using some interactive tools, where they’re all getting feedback at once. It was the only way I could see handling a group that large. If I was trying to do it as an online class, it would be effectively more in the form of a MOOC with very little interaction, either among the students or between me and the students.

Rebecca: One of the things that we both talked about before we started recording was how we both used a flipped classroom model to help with our synchronous session. So can you talk a little bit about how you did that and what students were doing outside of class.

John: This is actually, in many ways, similar to what I had done in a face-to-face class. Before each class, students would have some readings to work through. And I use the Lumen Learning Waymaker package, which is basically taking materials from a textbook, combining it with interactive multimedia, where they got to shift demand and supply curves and other curves around and see how they responded when they change parameters. And they read a bit in that online text and then they would work through some problems on it where they were allowed multiple attempts at those problems. I also created some videos with embedded questions that were at a somewhat higher level than the textbook readings, which was a little bit more challenging. And they were given unlimited attempts to work through those videos with the questions. And they also, outside of class, participated in discussion boards, where I asked them to relate what they were learning to things in the world around them, in their own lives and their own experiences.

Rebecca: …nice little inclusive teaching practice right there, right? …connecting students with their experience and making it relevant to them.

John: Right, because we know that students learn things most effectively when it has some salience, when they see the relevance to their life.

Rebecca: And the waymaker package, if I remember correctly, had some quizzing and stuff associated with that, and unlimited attempts, a version of retrieval practice there.

John: It’s a mix of things with unlimited attempts and limited attempts. So, the microeconomics Waymaker package is designed, and all of their Waymaker packages, for that matter, are designed, is that they start with a list of broad learning objectives. And they break it down at each module level to sub objectives. And they break those down into sub modules. So, in what would have been the equivalent of a chapter of a textbook, there’s usually two to four sub modules on particular aspects of that. And students work through that. And embedded in it, they have some review questions, some practice questions. And those they can take an unlimited number of times at any point in the course. Once they complete the module, they have a module quiz where they are limited to only two attempts at it. But they’re getting feedback on what they did well. And what they didn’t do well. It’s automatically color coded to indicate whether they mastered the material in one of the blocks of content in there. And then, if they take the module quiz, it will give them feedback on what areas they did well, and what areas they need to work at more. And they’re being directed back to the areas that they need to review. And there, they do have unlimited practice opportunities. And the other thing I did is I created my own videos that focused primarily on the topics that students generally find the most challenging. And in economics, that’s generally with either applications involving math or involving graphs. It was one module a week, and I would take the topics that I know, from past experience, they were likely to have the most problems with, I’d create my own videos with that. And I was using PlayPosit, which allows you to embed questions in there. Most of those videos I created were between five and 12 minutes in length. They would watch the videos and answer questions as they were going. And if they got one of the questions wrong, they could go back and replay that portion of the video and then try it again. And they were given unlimited opportunities for that.

Rebecca: I think you mentioned students really loved those opportunities.

John: At the end of the class, I gave them a Jamboard, which I know is something you’ve used more regularly, asking them what worked well. And there was very much universal agreement on the PlayPosit, as well as on the Waymaker aspects of the course. They really liked the fact that there were practice activities embedded right in their textbook, and that they could go back and try things over and over again until they mastered it. And it was giving them feedback on whether they had, in fact, attained mastery at every step. And it was a nice visual indication of what they’ve learned and what they still needed to work on more.

Rebecca: Excellent.

John: What did you do in your asynchronous components of your class?

Rebecca: Well, the balance of my classes, as I mentioned before, is a little wonky in that we’re supposed to spend more time in class and less time out. So asynchronously, I did a lot of independent stuff that students were not necessarily doing collaboratively. So this is where I had lecture videos that are recorded that were about the topics that they were going to be working on or introduce the component of the project that they were going to be doing. And then they also were completing things like LinkedIn Learning tutorials. And we also have access to D-Q University, which is a set of tutorials for accessibility, and teaches accessibility. So I took advantage of that package as well. And largely they were completing those kinds of tutorials, both of those have exercise files and that kind of thing that they can follow along with. They get little certificates. When they’re done completing there’s little quiz questions and stuff. So they were doing a lot of that kind of work asynchronously. They were also using Slack to communicate with their teams for independent things that they were working on that they needed to communicate out to teams when they were working on projects together. And I also use Slack as a place to have discussion. So like you, I had discussion questions that tried to make what we were talking about relevant. We were exploring design, specifically like web design and how they interacted as a consumer versus how they would interact as a maker and did a lot of observational studies. We also did some discussion boards that were really about design activities and things that got students off the computer. So they were just documenting what they did off-screen, offline. So things like listening to a podcast so that they didn’t have to be staring at a screen and what their takeaways were. They attended virtual conferences, which I guess was still on-screen, and did some sketching, like paper prototyping and some other methods that we like to encourage our students to do, just to kind of help balance the screen time a little bit for students. So that’s largely what they were doing asynchronously.

John: Were you having them submit some copies of that work in some way, or were they just reflecting on the work that they had done?

Rebecca: The little non screen activities were documented in a discussion, essentially, that we were holding on Slack, and then tutorials and things, they were just submitting their completed certificates. And so I broke down those LinkedIn Learning courses and things over multiple weeks. So they didn’t really submit those certificates until they were completed. But they were doing a little bit by little bit, but if they didn’t do the tutorials, they wouldn’t be able to do the projects or the actual work that we were doing of the class. So it was pretty important that they were doing those components outside of class.

John: Once your students were in class, what did you have them do in a typical class session?

Rebecca: The two classes I was teaching I handled a bit differently because of just the sheer volume of students in the bigger class, which was 25 students that were working on projects. They were working on collaborative projects, in teams of three, for the most part. And so what we would often do is show-and-tell’s or critiques in small groups. So let’s say there was two or three teams together that we would do a little critique with in a breakout room, while other teams were meeting and collaborating. We would also do things like come together to answer questions about things that they were working on, troubleshoot or whatever, and then go work on projects in breakouts for a bit. And then we’ll come back at a scheduled time. I also did one-on-one meetings with students during class time. So I’d set up things like a quiet work breakout room or the chatty breakout room. And students would pick the place that they wanted to go while they were working on projects. And then I would meet with them individually for critique, and often a lot of code troubleshooting is a lot of what I spend synchronous time doing. And students sometimes met with my TA to do the same thing, and with our small groups. I also did a lot of design challenges. And students really liked those and would like to do more to hold them accountable for the kind of material that they were learning outside of class or being introduced to outside of class in a low-stakes environment to test it out with some peers and troubleshoot. So I would pose a little design problem. And then they’d work in a small group to work on that problem in a very tight amount of time. They might spend 30 minutes… my classes are three hours long… or an hour, and then we’d come back and show them off or talk about different things. And I tried to make those design challenges fun and entertaining. So one of the first things we did, which worked really well to start gelling their teams that they were with the whole semester was designing an emoji for Slack that they used for their team. And they loved that assignment. It was partly about working at a small size, and so it was tied to some of the curriculum that we were doing, but it was fun. So they did that in a small team and then had to implement it. Later on in the semester, we did things like a 404 error page for their projects, which were just kind of entertaining. We tried to make them amusing, so that if you landed on a page, it was a good user experience. So things that maybe wouldn’t typically work on in one of my classes that were a little bit more fun, but really were emphasizing the technical and conceptual things that we were working on. The other thing that we use synchronous time for is I took advantage of our virtual platform, and I brought in alumni multiple times, and local designers multiple times and did little Q and A’s with them. Not every week, but every few weeks, or every couple weeks, I would bring in a designer for a 30-minute session. They’d introduce their work. And then students did a Q&A with them, which students really loved. And it broke up our time a bit and really gave them something special that maybe we didn’t always do in a face-to-face class that made the synchronous environment kind of special.

John: Excellent. That is a nice opportunity provided by Zoom that actually could work in the classroom too. But I think many of us just hadn’t really considered it so much. It doesn’t really matter where you are when you’re teaching in this sort of synchronous environment. So it’s very easy to bring in guest speakers and it’s something we’ve probably should have been doing more of in the past, but I think many of us will be doing more in the future.

Rebecca: So John, how did you use your synchronous time?

John: I had told students before each class session, what specific topics we’d be working on. And then most of the class time was spent asking him a series of problems of progressively higher levels of challenge. I basically adopted Eric Mazur’s clicker strategy of trying to find challenging questions where roughly half the class will get it wrong the first time and then letting them meet (in this case, I had the meet in breakout rooms), discussing it and coming back and voting again on it. And generally, you’d see a fairly significant increase in the performance after they’ve had that chance to engage in peer discussion. And that’s where a lot of the learning seems to happen when clickers are being used. I used iClicker. The only difference is students could not use a physical radio frequency clicker because they have a range of a couple 100 meters and students were spread out all over the world, I had one student in Egypt, I had students in South America and students spread throughout the country this time. So they needed to use either their laptop or a mobile device in order to do that. We discussed it as a whole class after they come back from the breakout rooms. And then I’d asked them to explain their choices. I generally have them use chat, and then I’d go through and correct any misperceptions they’d have. And I try to guide them to the correct answer by asking them questions, and letting them see for themselves why some of the answers were right, and some of them were wrong. And generally, that’s how we spent many of our classes. Initially, I was also using Kahoot! from time to time. They enjoyed Kahoot!, but I noticed a bit of a drop off when we were doing the Kahoot! sessions, because those were not graded. And with the clicker questions, they were being graded, and that tended to receive a somewhat higher level of interest. It was very low stakes, they got a certain number of points for an incorrect answer on either attempt, and they got a bit more points when they answered the question correctly. And initially, I was giving him three points for an incorrect answer, and five for a correct one. And they asked it perhaps that could be bumped up, because some of the questions were so challenging. And I did raise it. So they ended up getting four points for any answer, and five points for a correct answer. So it is extremely low stakes. So I tried to do a lot of retrieval practice in the class, where it started from essentially no stakes with the embedded questions in the reading, then it ramped up to in class applications of this, where they still get 80%, even if they got it wrong, but they had another chance to get it correct. And then they took that module quiz, and even there, they had two attempts at it. So if they made mistakes, they had lots of resources they could go back to and work on it. So I tried to set it up and provide them with many pathways to attain mastery of the content, and to encourage a growth mindset and to encourage them to recognize that people make mistakes when they’re learning and that there’s a lot of benefit from having those mistakes as part of your learning process. There’s a lot of research that shows that we learn things much more deepl if we get them wrong, when we first try it, we’re much more likely to remember it later on, then if we happen to get it correct, initially. In that case, we’re much more likely to forget it a bit later. And that was a bit of a challenge for students. But I think they eventually appreciated the fact that everything was fairly low stakes.

Rebecca: I think I’m seeing some themes in the things that, although we’re teaching very different classes in very different contexts, there’s some real big themes about how we’re using our synchronous time, and even how we’re using our asynchronous time. And so there’s an emphasis on peer interaction and establishing those peer networks, really enforcing or reinforcing things and dealing with muddy points. And then also just providing the encouragement and support like that low-stakes environment or trying to foster a growth mindset. So in my classes, I did the same thing. I was doing peer group work and trying to really get them to collaborate and troubleshoot together and they love that that… that was really valuable. I spent time doing live demos and troubleshooting, when there was a really troublesome technical component or something that they were trying to do that a lot of them were having trouble with, that they could ask me live questions. So that same muddy point kind of thing that you were getting to in what you were discussing. And then, finally, the growth mindset that you started bringing up, I’d also tried to do and, although I didn’t have a lot of low-stakes testing, or something like that, I set my projects up so they were done in sprints. So a long full-semester project was broken into multiple two-week sprints, where they would work on something, get feedback, and then could revisit whatever they did, and then add a new component to it. And so I did that throughout the whole semester. So there was a bit of retrieval practice, a bit of spaced practice in there, and certainly some fostering a growth mindset and the idea that you make mistakes and that’s how you learn. And I spent a lot of time… I don’t know if you experienced this too, John… but I experienced a lot of time in synchronous and saying like, “You can do this. It’ll be okay. And this is how the learning experience works.”

John: And I did have to do a lot of that, especially in the first few weeks of the semester, because they were not used to a flipped class environment. And they were not used to this notion of making mistakes and learning from mistakes as part of your learning process. Because most of them have come up through their elementary and secondary school system thinking that they need to memorize some things and reproduce it on exams. And they do well if they get high scores, and they don’t make mistakes. And that’s just not how we learn in general. And it was important, I think, to help remind them of that. Another aspect of the flipped class environment that we’re both using is that we let students learn some of the basic skills, the easy things that they can learn pretty easily on their own, from other resources. And we’re trying to focus our class time using essentially a just-in-time teaching approach where you focus on the things that students always have trouble. In a traditional classroom environment, what normally happens is students will learn the easy stuff in class where faculty will lecture them on basic definitions and basic concepts. And then it all makes a lot of sense until students try to apply it. And they try to apply it typically in assignments outside of class, or in high-stakes exams. And it’s much more productive if the students use the time outside of class to master those basic concepts. And then we hold them accountable for having done that somehow in class. And then we give them assistance on the things that they find challenging when they need it. Not after they’ve had that experience of a more high-stakes assessment in some way.

Rebecca: Yeah, I think what I found or that students really shared with me that was something that they really appreciated was that there was a lot of structured time to work on those difficult problems in class. This is true of my face-to-face classes too, but even maybe more so in this online environment where students were having a really hard time managing their time. I would allow time to work on a project during class… it was scheduled, but then there was a check in point later on in the day. You wouldn’t want to spend three hours staring at a screen on Zoom, like this makes no sense. So I certainly did not do that. And I don’t want anyone to think that I did that. But, I would do things like “Okay, we’re going to check in at 9:30. And then we’re going to do a little activity together. And then you’re going to have some work time to work on X. And then we’re going to come back at 11. And you’re going to show me what you did. And then we’re going to have a little discussion or do another little activity, and then we’re gonna come back again at 12.” And we would have a schedule where there was time to kind of come back. What I found is, over time, students often wouldn’t actually get off of Zoom. They would just turn their cameras off and their microphones off. And I would do the same if it was like a work time. And then when we all came back on, a lot of students would turn the media back on. That said, I, of course did not require that depending on where students were, I certainly had students that were in environments where they couldn’t turn their cameras on, or had really poor internet connections, we adjusted as necessary there, and we had a way to communicate in a much more low-tech fashion using Slack during class time. So if something happened with someone’s internet connection, or whatever, they could still stay connected with us and what we’re doing.

John: How did you assess student learning in your class?

Rebecca: My classes are all project based. So the majority of grades are built on projects, not entirely, we had discussion boards, and I had some collaboration things that they were doing, and they were evaluated on those things as well. But projects were the significant piece of the puzzle. And the way that I graded them was really just providing feedback about the kinds of things I was going to ultimately grade very regularly throughout the semester. So every couple of weeks, they were getting feedback on their code for my web class, for example, feedback on their design, feedback on their writing, not a specific grade, necessarily, but feedback on all of those elements that were going to go into the final project. And then the ability to revise all of those again and again and again and continue to get feedback on those.

John: Did you have your students engaged in any reflective tasks?

Rebecca: Yeah, that’s a really great question, John. I had reflection built in two ways. So at the end of each sprint, or kind of module in my class, they were working on two projects at the same time throughout the semester, so they’d work kind of two weeks on one project, two weeks on the next project and cycle back… that was so that I had time to give them feedback regularly. So that was part of my structure. But at the end of one of those modules, I had a reflection activity that I implemented using a Google form. So a few different prompts to think about what they got out of that sprint, goals for their next sprint, that kind of thing. And then I also had some big group reflections at different moments during the semester, I had one at the beginning, and a couple in the middle and one at the end. And I use Jamboard for that, which is a Google suite tool that has sticky notes, and is the same kind of way that you might brainstorm. So I use it as a way to collect reflections in sticky note form, essentially, virtually. And I would have a reflection question for folks to respond to or a couple of different boards with different kinds of questions. In the beginning, we did something called “hopes and fears,” which is something I’ve talked about before… setting up the class like, what are they hopeful that they’re going to get out of a collaborative project? What are they scared about? We find out that like, all the teams have the same hopes and fears. During the middle of the semester, what are some of the big takeaways that you’ve had? What are some things that you want to work on? What are some things that you’d like to see changed about the class and various themes bubble up on that. And then at the end of the semester, I asked questions like, “What was your biggest takeaway? What was the thing you were surprised that you learned? What is one recommendation of something you would change in this semester?” and “What is something that you want to continue learning?” and I got really useful feedback on what to change about the class but also, some really great themes bubbled up across the class, which really results in like kind of three or four things for each of those questions, which was a nice way to wrap up the end of the class and summarize for students after they completed that task. And one thing that I like about the Jamboard is that it actually ends up being anonymous. You can see people while they’re working on it, but it doesn’t keep a name with a sticky, ultimately.

John: So you can see who’s active in the board, but you don’t see who is writing which note.

Rebecca: Yeah, exactly. So that worked really well for me. How about you, John, were you able to build in reflection? I know you have such a big class. So it can be tricky.

John: I wasn’t able to do as much of that with my large section. But I did have them do that, to some extent in their discussions. For one discussion forum in both classes, I had them use a tool called Packback, which uses artificial intelligence to give students some feedback as they’re writing their prompts. And each week, students had to post a question related to that week’s material and they had to respond to at least a couple other people. But one of the nice things about Packback is it will check the cognitive level of the posts, it will give them some feedback in terms of grammar, it will also do a little bit of checking to see if the material has previously been posted. And it gives students some feedback, encouraging them to say more than “I agree.” And it also encourages them to document sources and to provide resources or references for the arguments they provide. And they get a score on that. So it takes a lot of the evaluation of that away. And so I monitored all of that. But it was something that seemed to function pretty well, just by the interaction between the users and that system. I haven’t really mentioned much about what I did in my online class. My online class uses many of the tools, but obviously, I couldn’t do synchronous, because that class is fully asynchronous. I couldn’t do the same type of instruction. But I had students do two other things in that class that provided opportunities for reflection, one of which was I had them work in a metacognitive cafe, low-stakes discussion forum, where they reflected on what they were learning and the learning process. And that gave them another way of making connections to their learning and reflecting on how well they were learning materials and what barriers they were facing, and also sharing effective learning strategies with each other. They were given some readings each week, generally on research-based learning practices as a primer for many of those discussions… others, they were just reflecting on what they’ve learned and how it might be useful in their life to tie it back to themselves. But the other thing I had students do is work on two podcast projects in that class. And in those they were taking what they were learning and making reflections about how that connected to the world around them. Many of them ended up being related to COVID and pandemics, but they were making some really good connections, and they were getting a chance to see how the material they were learning had some relevance in their own lives. And a lot of that came out in some of the things they were discussing in their podcasts. And they also did use Jamboard once at the very end of the term. But I also use Google forms a few times to have them reflect on the process of what was working, what wasn’t working in the class and what was working and what was not working in their own learning processes and what I can do and what they could do to help them learn more effectively.

Rebecca: I don’t know about you, but I was really surprised at how well synchronous learning actually went for me. I had some technical difficulties early on with my internet connection. It took me a while, but I got around to fixing that problem by hardwiring my internet and resolving some of those things. But I felt just as connected to my students as I would normally. I had a lot of interactions. And in some ways, I was able to facilitate those interactions a little more equitably online, because it wasn’t just the person who came to nudge me and stand in line and be the next person. Instead, I could really coordinate using waiting rooms and breakout rooms and really give everybody a chance to have one-on-one interactions with me, which I really appreciated. And I really did get to know all of my students quite well, which I was a little bit surprised about. And then, in an area where it’s really technicalaAnd we’re doing a lot of coding and things on screen, being able to share screens and take control of another person’s computer to fix things or show them how to do something was incredibly valuable. We use some of those kinds of tools in person. But it actually was, I think, in some cases more effective using this particular tool. So I was kind of surprised at how well some things worked. And I think that even when things go back to face to face, there’s definitely some components here that I would keep.

John: I’d agree. And I think students were amazed at how well some of those tools work. When in breakout rooms, they would be using the whiteboard features, they would be sharing screens, they’d be making the case, they’d be drawing on the screens, and that was something that would be much harder to do in a face-to-face environment. Initially, at the beginning of the class, I had some issues with chat being kind of flooded with irrelevant material, and I had to clamp down on that a little bit. But within a couple of weeks, they started actually using it very productively, and it provided a voice for all students, even those quiet students who would have otherwise sat in the back of this large lecture hall. They were able to type something in chat, after thinking about what they wanted to say before doing it, without being concerned about interrupting the discussion that was going on. And I think that was really helpful. And when I taught large classes with three to 400 students, there’s almost always 3 to 10 Students who have trouble not having side conversations when there’s other activities going on. And that mute option is kind of a nice feature and the ability to set their microphones so they’re all muted unless they choose to unmute… to have the default being muted until people click the unmute option… made it really easy. And I was amazed at how quickly they adjusted to muting and unmuting. By the end of the term there was maybe only once or twice a class where a family member or someone else would walk into the room and start talking. And then they’d remember, they had to mute their mics, and it was very rare. In a class that large, I was impressed by it… and working with students one on one, during office hours, it was so much easier to have students just share the screen and show you exactly what their problems were then to correspond with them with email, or even have them boot up their computer or you try to find what they were talking about when they came to your office. It was just much more efficient.

Rebecca: Yeah, I could actually see it. You can Zoom in, you can see what they’re talking about. I also found, and I was really floored, in this last week of classes, students were doing their final presentations,at how well they did develop facility with these tools. They’ve developed a lot of fluency in the kinds of tools that are actually very relevant to my particular discipline. It’s relevant to many disciplines. But designers use these tools all the time when they’re working with clients. And so it was amazing to me that we got through 15 presentations so efficiently. We didn’t wait for anybody to share their screen. They just knew what they needed to prepare, had it ready, they started developing slide decks really effectively, and could just do the things that they needed to do really efficiently. One of the last things I said to my class was like “I’m so proud of you just being able to do that. We didn’t have to wait for anybody today. That was amazing.” And so maybe a little bit of a blessing in disguise, you hate saying like, “Oh, the plague is such a great thing.” But they really did develop some useful skills and tools and they became more effective communicators. That was something that a lot of students reflected on and things that they didn’t expect to learn is how much better they became collaborators and just communicators generally… not just in person, like through Zoom or in text… like through chat in Slack.

John: Video conferencing is likely to be a part of their lives in the foreseeable future, especially now that everyone has adapted tp this mode, it’s very useful for them to learn how to use that efficiently. The one thing I do miss though, is seeing their faces in person and recognizing them. One concern that I have is, I’m hoping to be back on campus in the fall, there may be students that I work with who interact with me regularly, whose voice I would recognize or whose name I would recognize on the screen, but whose face I just wouldn’t recognize because a very large proportion of students just didn’t feel comfortable having their cameras on regularly, and I understand that. We’ve got a lot of students living in crowded living quarters or working with very poor network connections. But I do miss actually physically seeing them. And I had my last class session earlier today. And I encouraged them to stop by in the fall and just say hello.

Rebecca: Yeah, I mean, I agree that the physicality is certainly something that’s missing. But it was amazing to me how connected I still felt to all of the students at the end of the semester. And I think that they felt connected to each other too and they verbalized that, and also wrote that in their Jamboard reflection. So although there’s much to be improved, given this was the first time out and an experiment in many ways. I’m really thankful that I read Flower Darby’s book about Small Teaching Online because that actually informed a lot of my practices, even though it was synchronous, and a lot of her material was about asynchronous learning. It really did help me remind myself of things that I already knew that I needed to do, but to kind of make a checklist of things that I definitely needed to do as I was rethinking my classes for the fall. So thanks for chatting with me, John.

John: We always end with the question: “What’s next?”

Rebecca: I am sitting down to reflect on what worked and what didn’t work, to try to troubleshoot some things for the spring. And I’m teaching a class that’s brand new to our curriculum for the first time in the spring. And so we’re developing it for online synchronous, although ultimately, it’ll probably be a face-to-face class. We’ve had to re-conceptualize some of the things that we were going to do because of the technology limitations that students may have. If they’re online, we’re expecting that we might have a lot of students who are relying on their phones versus software and having access to high-end software packages or computers that can run them. So we’ve had to rethink things. But I’m pretty excited about being able to experiment with my students with all kinds of technology in the spring, but it’s definitely a puzzle that I’m currently starting to work on. How about you, John?

John: Well, I still have a lot of grading to do. But once that is done, one of the things I’m going to be doing is converting a textbook I had written in econometrics to a Pressbooks site, which will be a lot of conversion because it’s originally in LaTex, a typesetting language used for mathematical typing and I’m planning to create a lot of videos, I’m hoping to get many of them done over the break so that I’m not spending 15 or 20 hours a week creating videos as I was all fall. And I’m hoping to get a little bit further ahead of the semester this time, so I’m not doing as much preparation at the last moment. And we’re both going to be working on putting together a series of workshops in January for our faculty to help people prepare for whatever comes at them this spring

Rebecca: We’re just going to be really busy.[LAUGHTER]

John: I’ve never spent as many hours working on my classes as I have this semester.

Rebecca: I agree. There was a lot of startup costs converting to this modality, but I’m hoping a lot of that stuff I’ll be able to keep and reuse moving forward. Thanks, John. Always nice talking to you, John. We chat all the time. But it’s nice to sometimes hear about some of the thought process and things behind some of the decisions that you’ve made in your classes. So it was really nice to actually hear about how you did some of that stuff this semester. So thanks.

John: And I also appreciate hearing more about what you’ve been doing in your classes. We spend most of our time on podcasts talking to our guests and only mentioning little snippets of what we’ve been doing ourselves.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

155. Remote Proctoring

Faculty who rely on high-stakes proctored exams in their classrooms often attempt to replicate this approach in online instruction by using remote proctoring services. In this episode, Jessamyn Neuhaus and John Locke join us to discuss some of the issues associated with the use of remote video proctoring and suggest some effective and less problematic alternative methods of assessing student learning.

Jessamyn is the Interim Director of the SUNY Plattsburgh Center for Teaching Excellence and a Professor in the History Department at Plattsburgh. She specializes in the study of pop culture, gender studies, and teaching and learning. Jessmyn is the recipient of the State University of New York’s Chancellor’s Award for Teaching Excellence. She is also the author of Geeky Pedagogy: a Guide for Intellectuals, Introverts and Nerds who Want to be Effective Teachers. John is the Coordinator of Technology Enhanced Learning and an adjunct instructor in Communication Studies, also at SUNY Plattsburgh. He recently received his doctorate in interdisciplinary studies with a concentration in humanities and culture, and is currently working on a second historical novel.

Show Notes

Additional Resources/References

Transcript

John: Faculty who rely on high-stakes proctored exams in their classrooms often attempt to replicate this approach in online instruction by using remote proctoring services. In this episode, we discuss some of the issues associated with the use of remote video proctoring and suggest some effective and less problematic alternative methods of assessing student learning.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Our guests today are Jessamyn Neuhaus and John Locke. Jessamyn is the Interim Director of the SUNY Plattsburgh Center for Teaching Excellence and a Professor in the History Department at Plattsburgh. She specializes in the study of pop culture, gender studies, and teaching and learning. Jessmyn is the recipient of the State University of New York’s Chancellor’s Award for Teaching Excellence. She is also the author of Geeky Pedagogy: a Guide for Intellectuals, Introverts and Nerds who Want to be Effective Teachers. John is the Coordinator of Technology Enhanced Learning and an adjunct instructor in Communication Studies, also at SUNY Plattsburgh. He recently received his doctorate in interdisciplinary studies with a concentration in humanities and culture, and is currently working on a second historical novel. Welcome, John, and welcome back, Jessamyn.

Jessamyn: Thank you. Thanks for having us.

John L.: Yeah, thanks.

John: Today’s teas are:

Jessamyn: Just plain water for me. Gotta stay hydrated.

John L.: Grande decaf from Starbucks.

John K.: That’s an interesting tea.

Rebecca: Yeah. [LAUGHTER] I have a Scottish afternoon tea

John K.: …and I have ginger peach green tea.

We’ve invited you both here to talk about online proctoring services. As a result of the global pandemic, a lot of people suddenly had to shift from face-to-face instruction to remote instruction or online instruction. And many people who relied on proctored classroom exams are concerned about how to offer tests, and many faculty have been investigating the possibility of using remote proctoring services. What are some of the concerns associated with using online proctoring services?

John L.: Well, to start with, we are all trying to deal with the digital divide. And when you get into online proctored exams, that becomes a pretty big issue in that not all students have the equipment or the bandwidth to be able to participate. It helps to know what the process is. And basically, what we’re dealing with is a test that’s happening while the student is being recorded, both audio and visually being recorded. Usually, it starts out with a little intro section where you have to show an ID to prove who you are, show your space so that everybody can see that you don’t have crib notes on your desk, or there isn’t Albert Einstein in the corner of the room [LAUGHTER] telling you the answers to what you’re working on. And assuming all that goes well, then, of course, you’re taking the tests, usually an online test with a lockdown browser so that you can’t surf for answers anywhere else. It’s a lot of moving parts to make it work in the first place. And the big assumption is, number one, the student has the equipment necessary, and the student has the environment necessary to take a quiz like that. For instance, if you happen to be a student who lives in a very small apartment with a family, and you have brothers and sisters running through the room where you’re taking the test, because you’re at the dining room table, there are so many issues that come into play, not to mention just the fact that you may be embarrassed by your surroundings and don’t feel comfortable showing those surroundings to other people. So for me, that’s probably the first and most critical reason why I always talk to faculty and ask them to think about it before they actually devote themselves to that process. Other issues are, try as you may, there are always ways to get around these sorts of safeguards. And if there’s one thing we know, it’s that somebody who plans to be dishonest will figure out a way to be dishonest. Again, I try to get instructors to be a little more thoughtful with how they’re going to assess that learning is taken place in the first place. And that’s really where my friend Jessamyn has opened my eyes to many of the alternative ways.

Jessamyn: Yeah, there’s a lot of great resources that have been proliferating since the emergency pivot in response to this very question and suggestions, building on research that was already there, for how to assess student learning and in authentic and, as John was mentioning, equitable as possible way. I guess, just what I would add to that in terms of looking at it as a scholar of pedagogy, and taking messages like from James Lang’s book, Cheating Lessons, what do you want to foreground in your message to students in the class climate you’re creating, in the rapport that you’re building with them? The ordeal of the kind of proctoring software that John was describing, and that we were increasingly seeing problems with… the very first message you’re sending to students is: I assume students cheat, I assume students are going to be dishonest. I assume students don’t care about their education enough to try to express their learning as honestly and authentically as possible. And I guess what we, as what John and I both, were inviting faculty to consider when we were doing workshops this summer on this topic is: are there alternatives to this that send a more positive message and create a more productive class client and help you connect to students? Let’s not forget, at a time when everybody is anxious and overextended and fearful, we’re still in the middle of a pandemic. So, what do you want to prioritize as an educator?

John L.: Yeah, and exams are stressful enough as it is. So you add COVID on top of that, and then you add a technology that students aren’t used to. And it’s so much easier to choke under that environment.

Jessamyn: Yeah, an anxious brain is not a brain that can clearly and, to its best ability, express what it knows and show what it knows. All the information about trauma-informed teaching just reminds us that if every chemical and message in your brain is saying, “Run away from the tiger that’s hiding in the jungle,” there’s no room to: “Okay, move your webcam to show behind your ears that you don’t have an earpiece. Now take your laptop over to the door and show that it’s closed.” How is that not creating a prey state of mind with the predator waiting to pounce on you?

John K.: Each of the issues that you both talked about also have a very differential effect in terms of creating an inclusive classroom environment. People from high-income households are more likely to have some nice quiet space, are likely to be able to afford equipment that will work with proctoring software, while Chromebooks and most mobile devices will not work well with proctoring services. And also issues of anxiety and concern about being successful are also probably more likely to be experienced by students who are first- gen students who don’t necessarily have the same expectations of being successful based on their family environment and their social networks. One of the things that concerns me about all this is that the impact would be differentially imposed on students who are already at a disadvantage in terms of the quality of their prior schooling and their resources and their support networks.

John L.: Yeah, I couldn’t agree more. I’m not sure what to add to that, John. [LAUGHTER]

Jessamyn: I can jump in though. I had a thought. I’ve been reflecting… I can’t get it out of my head from a webinar this week that the Chronicle of Higher Education did a panel about the human element in online learning. And one of the panelists, Viji Sathy, mentioned that this crisis has really brought home to a huge new number of educators that we are teaching whole students… that taking into account all aspects of students experiences, their work experiences, family experiences, and these equity issues. So, it’s not that academic inequality is brand new to 2020. But, the awareness of it has really increased and the attention to it has really increased. And I think it’s being highlighted in ways that it’s just impossible to look away from. So this specific issue is touching on, I think, a bigger kind of reckoning that faculty are having on an individual basis, and as institutions. I see a lot more individual instructors really asking, “Wait, am I being inclusive?” The question is way more in people’s minds than I think it’s ever been, in my experience.

Rebecca: Related to that is the idea of accessibility too. With so much delivery in digital formats, the topic of digital accessibility is becoming much more prevalent in the forefront of faculty’s minds, whether they want it to be or not, it becomes something that everyone’s becoming more aware of. This same kind of software also imposes a lot of accessibility issues and barriers for students with disabilities, because a lot of them are not compatible with assistive technology and aren’t built to Web Content Accessibility Guidelines, essentially.

Jessamyn: And related to that, students with anxiety issues, who are struggling with mental health issues… the high-stress, high-stakes examination, in any format, is a challenge. But add to that the technology aspect of it, you are looking at assessment mechanisms that really isn’t being accessible and inclusive, it would not allow all your students to show you what they know.

John K.: One concern that I have about proctoring services is that faculty may see it as a simple solution that will allow them to use tests that they’ve created in the past. Many people have created very elaborate test banks in Blackboard and other places and then they expect that those questions can now be used, if they’re used in a proctored environment, not realizing that most of those questions have already been distributed to multiple sites out there and students would often have access to them, anyway. So I think that proctored systems can provide instructors with a false sense of security and as John mentioned earlier, they can be pretty easily defeated as long as students have devices that will allow them, for example, to do screen shares in the background underneath the proctoring service or perhaps have multiple devices where they can be looking up answers or using some other mechanism that won’t always be easily detected by the proctoring service.

Jessamyn: That’s a good point, and I know John Locke has addressed that issue. I mean, you don’t drill in on it, but when you’re talking to faculty, you often say, “And by the way, this is not a magic bullet, even if you go through all the trouble of setting it up.”

John L.: The idea that somehow having someone else proctor your exam is going to save you time…. That’s not how it works. These proctoring systems just flag potential incidents. You still have to go through and you decide whether or not those are warranted as cheating or if they’re just someone sneezed. So, between setting up the exam and then reviewing the flags, looking for false flags, I don’t know if it saves anybody any time.

Rebecca: I’m team workload reduction.

Jessamyn: Yes.

John L.: Yeah. [LAUGHTER]

Rebecca: So, what do we say to faculty who ask about replicating those high-stakes testing environments in their online environments.

John L.: I say: “Why?” I think that what would be more appropriate is to simulate the environment that somebody needs to perform in where they’ve acquired the knowledge in order to accomplish that performance. For instance, I taught a computer applications course years ago, and for the final exam…. I did have a final exam… but, I told them, “What I hope you get out of this class is to learn how to learn how to use software. So if you haven’t already learned how to learn to use software, now’s your chance. And when you’re out in the real world, you will have the software manual, you will have the person in the cubicle next to you, the only thing you won’t have is me. So, unless you have a question about a specific question on the test, don’t talk to me, as far as anything else that gets you to accomplish the goal, go for it.” If you’re studying to be an ER doctor, perhaps you do need to have the pharmaceutical manuals memorized page by page. [LAUGHTER] But most of us aren’t working in that kind of stressful environment. So, there are better ways, maybe project-oriented ways, to assess that that learning has taken place, that those skills have been received or learned and received.

Jessamyn: I try to assume best intentions on the part of all faculty. And I know that many of my colleagues who expressed that sentiment exactly, like “How can I make sure they’re not cheating?” …they’re not saying that because they’re evil, like “Mwah, hah hah hah, those bad students…” No, they really are concerned about student learning. So, what John and I did was really to frame this as an invitation to faculty, an invitation to think creatively about assessment, authentic assessment, to really be able to measure student learning, but maybe also rethink what you thought and assumed about assessment. And here’s a big bonus, maybe grading it could be less painful. If you are trying something new, something that’s a little bit more creative, that might help you as well in your end. So, that’s been how we’ve been addressing it here at Plattsburgh.

Rebecca: What are some ways to do that assessment, maybe in a class that doesn’t work well for project-based learning. Maybe it’s a bigger section class, or maybe it’s more foundational information that doesn’t lend itself as easily to project-based learning. What are some alternatives?

Jessamyn: There’s always small, lower stakes, regular quizzes. So instead of one big, huge exam, having smaller quizzes along the way. That’s just one off the top of my head… an easy one. John?

John L.: Yeah, well, especially in this environment, discussion forums are really, I think, underutilized. There’s no reason that you can’t build a rubric around a discussion forum and spell out your expectations to students and then hold them to them and grade according to those. Again, it’s taking the student higher up that Bloom’s taxonomy ladder than just memorizing and regurgitating information. It’s causing them to react to other people’s comments within the discussion forum, to assimilate the knowledge that they’ve already accumulated, and to create new and different responses based on that immediate situation. And, the advantage to that for slow thinkers like me, [LAUGHTER] is that you don’t have to be quick on your feet. You’re not the student in the back of the room with his hand up saying, “Well, never mind, you covered that five minutes ago.” It’s kind of an equalizer. I wouldn’t say “Have a discussion forum as a final exam,” but it’s another part of the scaffold to assess that learning is taking place throughout the semester.

Jessamyn: I think there’s a lot of potential for open-book exams as well. In fact, I have used open-book exams for a long time. And, in large part, that is because I really wanted my students to learn, and I wanted to be able to grade an exam very rigorously. So saying, here’s a question you can answer with an open book, and, yeah, you might even talk to someone else about it. But then the final product is an essay question, or it could be a presentation, it could be a sort of annotated bibliography. There’s lots of ways it could go as an open-book exam. But then when I go to assess it, I know that you have the material in front of you. So, I am going to really drill down here, like, “Do you really understand this concept? Can you show me that you understand it?” Because I know you can look at the basic definition in the book that’s open in front of you. So, now you have to show me that you really, really get it, you have to use it, you have to apply it, whatever it is.

Rebecca: What about STEM-oriented examples? A lot of the things that we’ve talked about work really well in the humanities and the arts. How about some things that work well in math and science and other STEM fields?

Jessamyn: So, I’ve been trying to do a little reading in this area. I’ve been hearing from some faculty in this area. So, in an online lab setting, being able to complete the experiment in the correct way, in the scientific-y way… [LAUGHTER] …that could be one way to assess learning… doing something like a fact sheet. So the final product is how you’re assessing the student learning. But again, you could be measuring the application, the correct way to do XYZ in a kind of fact sheet format or a PowerPoint slide or a poster presentation.

John K.: One type of thing we sometimes recommend for people in the STEM fields is that, if they are going to use multiple choice, one way of dealing with this is to use some algorithmically generated questions so that each student gets their own version of the question. Now, the solution procedure may be the same, but for at least low-level skills, that can help to deter some academic integrity issues.

Jessamyn: Student-generated exam questions could be another way to go. If you really understand the material, you’re not just regurgitating memorized material, but if you really understand it, then you should be able to help someone else understand it. And one way you could assess that would be “What are the 10 best exam questions?” …something like that.

Rebecca: Another idea that I’ve heard from people more in the STEM areas is the idea of creating some sort of resource that explains a topic to a non expert audience. So, maybe it’s an experiment or something that you can do with kids, or just kind of generally to someone who’s not in the discipline and get them to grasp whatever it is that you’re trying to assess.

Jessamyn: Yeah.

John L.: This might be going out on a limb for a STEM environment, maybe we could call it STEAM, because there is an artistic bent to it. But, for instance, in an accounting course, if there’s a particular accounting procedure or process that students have to prove that they understand it, they could write a short story, “a day in the life of the accountant to the New York Yankees” or something… and totally fictional, but covering each step in the process that has to be accomplished. And as an instructor, I would love to read something like that rather than checking off right or wrong on a test sheet.

Jessamyn: I’m thinking too about something like following up Rebecca’s suggestion, and increasing accessibility, you could even have students creating resources like that in a variety of formats. It could be a poster, could be a podcast, could be a video, could be a live presentation… You could do something like an oral exam… something like that.

John K.: One of the things I’m doing in my small class of 60 students is having students create podcasts. Unfortunately, that doesn’t scale as well, in my class that’s closer to 300 students. So, I’d really like to do more open pedagogy projects. It’s just, in large intro classes, that’s a bit of a challenge.

Rebecca: John, you have some experience using algorithmic questions, too, as a way of assessment, right?

John K.: Algorithmic questions can work very effectively, in at least making sure that students can use the formulas appropriately, which is a basic skill in many STEM classes.

Jessamyn: What I would like to see is more faculty really having these discussions and swapping these ideas, like on a national scale. I think that the learning curve has been so high for so many instructors in so many ways. Like, not just, “I’ve never even visited the learning management system, and now I have to use it.” Not just that. But, coming to terms with the emotional aspects of teaching and trauma-informed teaching in the midst of, possibly, “I’m at home and I’m supposed to be overseeing my children’s education” or simple childcare issues. All these things are overwhelming so many instructors just day-to-day life. And then on top of that, “Oh, rethink something you’ve used forever. The thing that you relied on from day one, and that you did so well in graduate school… hey, that’s not gonna work.” That’s hard. That’s tough. So, the more sharing of ideas we have, and the more spreading of good possibilities for assessment, the better. And I sent you a list of some of those resources I’ve been providing. They are starting to be generated, especially at university teaching centers and in people’s blogs and essays and such. But, I think the more it just becomes a broad conversation about “What can we do? How can we, in this situation, assess student learning in new ways and recognizing it’s new for us, too.”

Rebecca: Bill Goffe, in our episode 154, Sharing Disciplinary Pedagogies, also offered a way to get people to collaborate across institutions on some of these kinds of things using a simple Google Sheet. So, we’re all kind of forced to be on line in some capacities now, maybe more than before, but maybe that’s also opening some doors for collaboration that haven’t been there before, either.

Jessamyn: I hope so. I mean, John Locke and I, both of our centers had not been collaborating in the past. So, spring of 2020, was like this kind of completely perfect context for us to send a message to the university, the Center for Teaching Excellence, and Technology Enhanced Learning, we work together, and because people needed us both. So, in that sense, I won’t say silver lining, there’s no such thing right now, but it was a unique opportunity for these two very small centers on campus to collaborate.

John L.: Yeah, in fact, I’ve accidentally come up with a tagline that is starting to appear at the bottom of my emails to faculty. And that is, “you are not alone.” They never were, but it’s much more important for them to realize. In fact, I was working with a professor last night who was having some difficulty in the learning management system. And about 10 o’clock, I sent him what I thought was probably the solution. And I didn’t hear back. So, this morning, I sent him an email and said, you know, “How did it work out?” And his response was, “I’m sorry, I haven’t even gotten to it yet. I’m sorry.” And I said, “No, you don’t have to apologize to me, I just want you to know that you’re not alone, that I’m trying to help you. And I’m not going to let go until I know your problem is solved.” And that sort of community approach to learning in general, and what we’re all going through, I think is helpful. If you know that I know I’m struggling with this I’ll bet someone else is too and, maybe between us, we can figure it out. If more people can adopt that thought and not feel that they’re infringing on someone else’s time, I think we’ll all get through this to whatever the other end looks like.

Jessamyn: That was one of the first things that John Locke has said to faculty who wanted to use this remote proctoring system is “Don’t make your life harder than it has to be.” All the student issues aside, and equity and trust and accessibility, but it’s such a pain in the ass. It really is hard to use. And I’m not just talking to the student end is terrible, but from the instructor end. It’s such a pain to set up and he shared with me, sometimes someone will approach him, “Can I set this up,” he said “Okay, but you have to do bla bla bla bla bla, then this and this…” and they’re like, “uh, maybe I’ll rethink this.” LAUGHTER] I mean, let’s try to make our teaching a little bit more joyful, if we can. Let’s try to make it a little bit more creative, for our sake, if nothing else,

John K.: It can be a lot more fun listening to podcasts students create, listening to their videos that they create, looking at documents they create, or infographics and other things, than it is reading a pile of exams, or writing up multiple choice exams.

Jessamyn: For students, too. Conveying their knowledge in a different way. It’s so good for their brain. That’s why I’m always reassuring students, when I’m asking them to do non- traditional assessments, which I mostly use (even before all this). Our students are very traditional in many ways, and they get really nervous when I say, “Okay, so you’re gonna write a short story, you’re gonna do a poster.” And they say: “Wait, what? I’ve never done that before.” Or “ I don’t know, I don’t know if I can do that successfully.” And I’m constantly telling them, “This is you conveying your learning, your skills, your knowledge in a new way, and it feels challenging, but you could do it and it’s great for your brain. It’s like calisthenics for your brain. You’re presenting what you know, just like you would in a traditional research paper or a traditional exam, but it’s in a different format, and that’s great for your thinking in all ways.”

John K.: We always end with the question, what’s next?

John L.: What’s next? I’m waiting for that chip to be implanted in my head so that I won’t have to show you my assessment, you’ll just be able to download it. [LAUGHTER]

Jessamyn: John, what is your next book project?

John L.: My next book project… I’m writing a novel that’s called “Defending Eldorado” and it takes place in South America, about 50 years after Columbus, where a bunch of colonial powers are trying to find Eldorado and the native South, Central and North Americans are doing their best to make sure they don’t find it. And since we never did, obviously, they were successful. Spoiler alert. [LAUGHTER]

John K.: You mentioned that you had just completed a book. What was your most recent book about?

John L.: Ah, my most recent book was actually the prequel to the current book, a nd that was about a group of disillusioned European scholars who left the Academy. They were humanists, they left the academy because it was being run by scholastics. And they decided to find Thomas More’s Utopia, which leads them to the New World, and hilarity ensues. Not really, but… [LAUGHTER]

Rebecca: How about you, Jessamyn?

Jessamyn: I’m headed, coming up, very shortly, I think everybody here is familiar with it, the SUNY Faculty Developers Conference, it’s going to be online and I’m doing a poster there about a series of events that John Locke and I hosted over the spring for faculty. So, that’s coming up next month. I’ve got some speaking things coming up. I’m really excited to be speaking at the Lilly Online Conference in November, and I am reading chapter submissions for an anthology project that’s contracted with West Virginia University Press in their Teaching and Learning Series. It is an anthology of insights into effective teaching and learning from women, marginalized, and underrepresented faculty. I have some fantastic submissions… so many good ones. So, that’s been a really great thing I’ve been working on right now. It’s fun.

Rebecca: Well, thank you both for joining us…

Jessamyn: Thank you.

Rebecca: …and we look forward to your future work, for sure.

John L.: All right, thank you.

John: It’s great talking to both of you.

Jessamyn: Nice to see you both. Hang in there, SUNY Oswego.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Brittany Jones and Savannah Norton.

[MUSIC]

102. Team-Based Learning

A large body of research finds that active learning approaches result in larger learning gains than traditional lecture approaches. In this episode, Dr. Kristin Croyle joins us to discuss how she transitioned from  explore using interactive lecture to collaborative learning, and then to team-based learning. Kristin is a Psychologist and our new Dean of the College of Arts and Sciences at SUNY Oswego.

Show Notes

  • Johnson, R. T., & Johnson, D. W. (2008). Active learning: Cooperation in the classroom. The annual report of educational psychology in Japan, 47, 29-30.
  • A discussion by Dan Ariely explaining why asking for shorter lists of positive features in a relationship can engender positive feelings appears in this March 24, 2014 video clip.
  • Michaelsen, L. K., Knight, A. B., & Fink, L. D. (2004). Team-based learning: A transformative use of small groups in college teaching.
  • Team-Based Learning Cooperative
  • Epstein Educational Enterprises, What is the IF-AT?
  • Sweet, M., & Michaelsen, L. K. (2012). Team-based learning in the social sciences and humanities: Group work that works to generate critical thinking and engagement. Stylus Publishing, LLC.
  • Croyle, K. L., & Alfaro, E. (2012). Applying team-based learning with Mexican American students in the social science classroom. Team-based learning in the social sciences and humanities: Group work that works to generate critical thinking and engagement, 203-220.
  • Deslauriers, L., McCarty, L. S., Miller, K., Callaghan, K., & Kestin, G. (2019). Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences, 116(39), 19251-19257.
  • 74. Uncoverage – David Voelker – Tea for Teaching podcast episode discussing the uncoverage movement in history, March 27, 2019

Transcript

John: A large body of research finds that active learning approaches result in larger learning gains than traditional lecture approaches. In this episode, we explore one faculty member’s transition from using interactive lecture to collaborative learning, and then to team-based learning.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Today our guest is Dr. Kristin Croyle. Kristin is a Psychologist and our new Dean of the College of Arts and Sciences at SUNY Oswego. Welcome, Kristin.

Kristin: Thank you.

John: Our teas today are:

Kristin: Earl Grey

Rebecca: I am having Mama’s work tea, because Ada made it this morning and she calls it work tea, which means she pulls the tea bag tag out and puts it in the big cup. Also, it’s just my normal English Afternoon. But, that was a better story.

John: And I’m drinking Spring Cherry green tea.

Rebecca: We invited you here today to talk about collaborative and team-based learning in your classes. But before you do that, you’ve noted that you had a strong preparation in teaching before you got started. Can you talk a little bit about that. We’ve talked a little bit about that on the show before and how a lot of faculty aren’t prepared…

Kristin: Um hmm.

Rebecca: So, could you talk about how your preparation may be informed what you’ve done.

Kristin: My graduate program, I went to the doctoral program in clinical psychology at the University of Montana in beautiful Missoula. And that program takes the preparation of their grad students very seriously, but across several areas not just in clinical work and research, but also knowing that some of them are going to end up in positions in which there will be teaching. So, while I was there, that very first semester I was brought in, they had a structure for teaching their introductory psychology classes where graduate students were assigned our own classes where we were the instructor in the classroom, but we had a supportive network around us. So, the syllabus was already there, the textbook was already there. We collaborated in writing tests. We had a structure of TAs that supported us and they would have recitation sections in which the TAs also received development. And we joined in that so we could see how more hands-on kind of things could be done with students in smaller groups. We even assigned our final grades together. And some of those pieces are pieces that are areas of skill that people don’t often think about developing. So, that first semester, all I had to do was think about working within the structure: How am I going to handle the day-to-day teaching and learning in the classroom? I didn’t have to worry about course design because the course was already designed in front of me. And I also didn’t have to, at the end, think: “When you assign grades, is that rigid? Do you really have to follow the exact, you know, 90/80 that it is in the syllabus? Or what if there are natural breaks around 88 or 89? Is it okay to flex that? What kind of power does an instructor have that is fair to students and evaluation?” I got to do all of that in a collaborative setting with a very experienced faculty member as a guide. There was also a credit-bearing course for teaching psychology that we were encouraged to take… which I really enjoyed. And then I was given opportunities to function more independently. When they needed a stats teacher over the summer, and they knew I was living there over the summer, I got to teach on an adjunct basis, but still with the support of faculty around me. So kind of putting students in the deep end, but with a high level of support around them, I felt very prepared when I was done with the graduate program to enter into an assistant professor position. And I still appreciate the preparation that they gave me.

Rebecca: I think with the preparation like that you’re probably far more willing to experiment and do new things as a faculty member too and to maybe even break away from what faculty around you are doing. Do you find that to be the case? Or were there other faculty doing some of this collaborative work in the department that you were in?

Kristin: Yes, and no. One of the experiences I had at my previous institution, which was the University of Texas – Pan American that then transitioned through a merger to be the University of Texas Rio Grande Valley. I was talking with a colleague in another department about the kinds of things we were doing in the classroom. And I still remember him saying “Oh, I didn’t know you could do that in the classroom and that was like teaching.” He had a very restricted idea of what teaching was, and what would be acceptable to colleagues, which he had never had the opportunity to test with other people around him. And that was something that I arrived from day one… that you talked about your teaching… that you can do many different things in the classroom. And it’s all teaching, as long as you are trying to work with students to create a learning environment and they are learning, then it counts as teaching. So I did come in with a much more flexible idea, then certainly some of my colleagues who hadn’t had an opportunity to ever have those discussions. And of course, some people are hired into departments in which those discussions don’t ever happen, so they may persist with those misconceptions for many years.

John: Or throughout their entire careers at times. [LAUGHTER] The scaffolding that was provided is really nice, because we’ve talked to a few people who’ve been in teaching training program or had some training in graduate programs. But usually, it’s not quite as structured as that and that’s a nice feature.

Rebecca: Yeah, I came from a program like that, but it was like very front loaded. It wasn’t that ongoing…. So I felt a lot more prepared, because I did have a lot of those experiences, but I didn’t have that same kind of supportive network throughout. Which is incredibly valuable.

Rebecca: So, you want to take us through what some of your collaborative experiences have been in the classroom and the ways that you set up some of the team-based learning exercises, maybe starting with what are those?

Kristin: Sure. So kind of the way that I journey through my teaching, particularly when I was an assistant professor, I felt comfortable in the classroom, but I didn’t feel expert. I felt like I was still trying to figure out what was going on, which is a perfectly fine way to be and a good state for learning to occur. So I felt like I was a talented lecturer, like I can engage students. I teach in psychology, I also think psychology is naturally very engaging, but part of that is because I really love the field. So, I felt like I could engage students and that they would listen and that they would be interested. But I started to become dissatisfied that there was always a core of engaged students and I had no idea what was happening with the other students in the class. And then sometimes I would be disappointed when we have tests or homework. Everyone said they had no questions. Clearly that was wrong. I was wondering how do you engage the majority or all of the students in their learning so that they aren’t coasting through class believing that they understand until they really don’t. And then I also felt like I was kind of fooling myself into thinking that students were with me when they were not with me. So I had an opportunity at that time to do some intensive cooperative learning training along the model of Johnson and Johnson collaborative learning. And that model from the University of Minnesota, it focuses on the importance of cooperation in the classroom, and that in cooperative settings, students learn more, develop a stronger sense of self efficacy around their learning; that they together are able to achieve more than they would individually. And it also has impacts on retention… that if students are feeling like they are individually known and valued in the classroom by their peers, they’re more likely to continue showing up to class and to develop relationships outside of the classroom that supports them along the way. So through that training, it was intensive, it was like eight hours a week, one day set for several weeks. The very first day, I could see what a difference I was going to make in my classroom. So, for example, I was using group assignments in class and they had all the same disadvantages that group assignments and most classes have, because I had no idea how to structure the group work so that it would be successful. I was doing group work to save me grading time, honestly.

Rebecca: That’s why a lot of people go to group work.

Kristin: Yes! Without understanding that all I had to do was some structural changes, and then it would actually be effective for learning as well, instead of just saving me grading time. In that cooperative learning training, I learned how to structure intensive group work that could be the length of an entire semester, or it could be the length of a single class day. I learned how to structure less intensive moments of team time. So how do you do a think-pair-share that works versus how do you do a think-pair-share that doesn’t work very well. So, that within the course of that training… actually just within a few days… I suddenly had, instead of 10% of the students in the class engaged on a daily basis, I had 100% of the students engaged on a daily basis. So, that was a huge breakthrough and I continued that way for several years.

John: What were some of the structural changes that you made that did lead to increased engagement?

Kristin: So, the cooperative learning approach of Johnson and Johnson, is kind of theoretically heavy, in the sense that they outline the pieces that are necessary for strong collaboration to occur. And then they turn it over to you as the instructor to say, “How do I build those pieces in?” So, for example, they emphasize positive interdependence as one of the essential components of cooperative learning… that when you create a group and a group activity for them to do, the activity has to be structured in such a way that each person is necessary to contribute. You can’t structure it in such a way that you can have three people talking when one person is only needed, and there are specific recommendations on how do you structure it so that everyone is needed. At the same time you have to build an individual accountability as another required component, so that, even if each person is needed, people can still slack off, say, “Yes, you all can’t do as well without me because you need me, but I don’t really care about what is happening here.” There has to be a level of individual accountability that’s also built in. Along with that, some of the skills that I thought were most important, they build an emphasis on group processing and social skills, so that if you have people consistently working together in class, they may not have developed the social skills to do that effectively, especially over time. You can work with someone for two minutes on a think-pair-share and really be bad at social skills. But, if you have to work with them over an extended period of time on a project and things are going south in terms of group conflict, it’s the instructor’s responsibility to help them to develop the social skills to work together. For example, on the first day of class, when I first start having students talk to each other so that they know that’s going to be a pattern in the class, I give them something quick to talk about. And I say introduce yourself to the person next to you… spend two minutes talking about this. And then I’m going to ask you about what you talked about. And then I run around the class real quick… pair up people who aren’t participating, introduce them to each other so that they understand this is a part of the class. So, then I follow back. So, what pieces are important there? …that I explicitly instruct them, you turn with your body… you actually make eye contact. And I will point out as people first start doing this, look at these two people, they are looking at each other, because many times students won’t do that, and it’s very hard to have a cooperative interaction if you don’t make eye contact… and I will say, “Who was the person you talked to? Tell me their name.” So they understand that I was serious when I said, introduce yourself, tell me something about them and that there’s individual accountability through just random calling on… that they need to participate in the cooperative portion. And then there’s also the self-reinforcing aspect of it that five minutes later, when I say to talk about something else, they realize they already know somebody in class, they have a connection. The next day, when I come in, they’re not quiet, they’re already talking to each other, they’ve created those connections.

John: A nice thing about that, too, is for people who are uncomfortable talking about themselves in class, having one person tell you something about the other person, it’s a little bit less pressure, it’s a little less revelation to the whole group. There’s some evidence that that type of thing is more effective in providing a more comfortable environment.

Rebecca: Kristin, can you also talk a little bit about a specific example of a cooperative activity where all of the members are held accountable, and all have a role? …just to provide an example for people who have less experience.

Kristin: So cooperative learning can be divided into informal and formal cooperative learning. Informal cooperative learning tends to be much shorter activities that can be done kind of on the fly if you already have an idea in your mind of how you might want to do that. Formal cooperative learning tends to be more intensively structured… longer-term activities. So that could be a single class session. If you’re going to do an activity that takes an hour, that would be more formal… or if you’re going to do something that takes an entire semester. The pair-and-share that I just talked about is an example of informal cooperative learning. Something like a jigsaw classroom activity can be structured as a formal cooperative learning activity. And it already shares almost all of the components: there’s individual accountability, because each student is given a specific role. There’s also positive interdependence, because the success of everyone depends on each person doing their role. So there are ones that are already structured with a built in component. The pieces that aren’t built into something like a jigsaw classroom activity, would be the group skills and group processing, and the ways that you can build that in. You can, for example, ask groups to reflect on what went well. I typically emphasize that more than asking them to reflect on things that went poorly, because asking to reflect on what went well tends to maintain a positive atmosphere, but also helps them to cover both bases at the same time anyway.

Rebecca: …or realize that my list for what went well is not very long… [LAUGHTER]

Kristin: Right. So, a common group processing thing I would have students do after their first more lengthy or more formal cooperative learning activity would be: list three things that your team did well together and one thing that you could improve on. And another thing I might ask them to do is to provide positive feedback to each member of the group at the end of the activity. And the kind of feedback that they provide is usually pretty specific, and helps to shape their behavior throughout the rest of the semester. So when they say things like, “I like it when you disagreed, and you said that this other thing would be a better way to go” that provides important feedback, and it helps to encourage better processing going forward. But I will go around and give individual social skill feedback too. But it’s usually things like, “Oh, I see you’re sitting so far away from your group, I’m not sure they can hear you, let’s scoot your chair in so that they can hear you.” Or I might ask, “Oh, do you know this person’s name next to you? What’s her name?” …and we’ll make sure that people maintain the social and cooperative connections that enable to do that kind of good group.

John: Just as an aside, it’s useful if you’re asking about things that went well, to keep the list fairly short. I’m reminded of a study that Dan Ariely talked about where they did a controlled experiment where in one case, they asked people to reflect on three things they liked about their partner and another case to list 10 things I liked about the partner, and then they surveyed them on the quality of the relationship. And those who were asked about three things generally rated their relationship with a partner fairly high. But when they were asked to come up with 10 things, they struggled with that and they rated that relationship lower. So keeping the list short…

Kristin: Right.

John: …is really good so you don’t…

Kristin: There’s kind of analogous thought about keeping things like gratitude lists. If you list too much stuff, it can have a negative effect, because you start to identify things that you really don’t think are that important, and it makes you think the whole thing is less important.

John: And if you want to get the opposite effect, ask people to list 10 things that were bad, and then they’ll struggle beyond the first few. You talked about having continuous relationships or persistent relationships with collaborative learning. Did you try to keep the group relationships consistent for the same groups throughout the term? Or did you vary that?

Kristin: I varied it. There are some good data to suggest that in collaborative learning… they refer to them as base teams… that base teams have a persistent positive effect, particularly on things like student engagement and retention throughout the semester and throughout the year…. that you have a team that is expecting you every day. But when I was doing cooperative learning, I didn’t restructure my courses. I restructured the day. Does that make sense?

Rebecca: Um hmm.

Kristin: So I didn’t have a reason for base groups. And I felt strange imposing them on the students without a reason. Besides, they would maybe be socially a good idea. I had to completely rebuild my courses from the ground up before I started using base teams. And that’s when I transitioned to team-based learning.

John: …and in team-based learning, persistent teams are recommended as part of the process.

Kristin: Absolutely.

John: Could you tell us a little bit about this transition to team-based learning. What prompted you to introduce that? …and how it worked?

Kristin: So I was happy with how courses were going. People were interested and engaged. I had students telling me, “I know every single person in this classroom.” and when you’re teaching a class of 30, or 40, or 50, that’s unusual. “I know everyone in here, I feel really supported.” I feel like things were going well. But I was unsatisfied with what I was teaching. I wasn’t clear, in my own mind, about what persistent learning outcomes I wanted for my students. I had not sat down and really thought through if I were to follow up with a student in a year or five years, what would I want them to recall from this class? What would I want them to be putting into use in their lives or in their careers? I had never thought that through. And I was fortunate enough to run into team-based learning at that time, right as I was primed to start thinking about this questions. Team-based learning originated by Larry Michaelsen. He was coming from the perspective of enrollment increases. He had been assigning some pretty challenging work. He was a faculty member in business. And as his course enrollments increased, he started to wonder how can you maintain the same kind of interesting, really challenging in class… by case work, for example… with a large enrollment. So he developed team-based learning to address that piece, but it also requires you to completely rethink the design of the course. And to start from the course outcomes: “What do you want the persistent outcomes to be?” …and then structure the course forward in that way. So in team-based learning, after you make a decision about your course outcomes, and what you really want students to be able to do, then you structure the course in a modular fashion. And each module has certain steps. So the beginning is student preparation, then when they come into class, you test. You say, “okay” …and it’s called the readiness assurance process. So you want to know what students are ready to do after they’ve individually prepared, and what they’re not ready to do. So they prepare, they test. And then, since it’s a team focus, they also test as a team. After that you have a good idea as an instructor, what are they ready to do? What are still the fuzzy areas? What do they really not get at all? What are their competencies as a team already, even if every individual student doesn’t have it, and then you can do some corrective lecturing, basically, so many lectures that fill in some of the gaps. And that’s all part of the readiness process, because you’re getting them ready to do some interesting application work in class. And the rationale for that is… and actually what I had been doing prior to that, was giving interesting application material to work on at home individually, while doing lecture and cooperative learning in class. But the interesting application material was actually the heart of the course, and the much more challenging piece. So it was better to bring the hard piece where they needed support into the classroom. And the piece they were ready to do, which was to do their own self study back into their own lives. So you do this readiness assurance process to make sure they are ready for interesting application, and then the majority at the time for the module you spent on application. Doing that after I had already worked with cooperative learning was really helpful, because all of that application work is done in a team setting. So when you already have some experience with how to build teams, how to maintain and develop their social skills, that’s really, really, really helpful. That’s a short version.

John: One of my colleagues, Bill Goffe, who was on one of our very early podcasts, noted that when he gave the group test, the performance always went up significantly, so that they could see the benefits of the peer discussion that was part of that. And he was really impressed with it. And he noted that, oftentimes, if a student didn’t show up for class one day, they get a hard time from their classmates from the group because they let the group down. And he said his attendance had never been better than when he was using a team-based learning approach.

Kristin: Absolutely. And a lot of people who do team-based learning, use the same methodology for doing the team testing, which is honestly really cute. It’s a scratch-off form. And the scratch-off form is used so that the team gets immediate feedback on each option. So on any particular item in a multiple choice test, if they want to select “B” they scratch off “B.” If it’s not there, then they continue to scratch until they get the right answer. For one thing, they love it. But also they are getting immediate team feedback. If this person is not speaking up, if they say I think it’s “B” and then they stop advocating and then it turns out to be “B” later than the team immediately knows, by the time they get to the next question. “Okay, we need to incorporate more feedback from all of our team members, wait a minute, this person who’s not speaking up actually has a lot to say.” In the course of just a few multiple choice questions, it brings their team development forward leaps and bounds. And they kind of have fun with a scratch off, which is also a bonus.

John: And it also gives them incentives to come prepared and to listen to other people in ways that they might not otherwise.

Kristin: Yeah, and their team will give them grief, if they say “Oh, I don’t know, because I didn’t read,” their team members will be like, “But we are depending on you, you need to read, we all read.”

John: And it also gives them a little bit, perhaps, of improvement in metacognition because they’re getting that immediate feedback, and it’s being coupled with the reactions of the peers. So if someone was insistent on a wrong answer, and they dominated that discussion, they might be a little more careful in the future and more willing to listen to the other people and reflect.

Kristin: Exactly, and it doesn’t have to wait till next week, it can happen right away. Right on the next question. The team application activities are also structured in a particular way. In team-based learning, they talk about the four S’s for the application activity, the first one is that you have to select a significant problem. So what they’re working on is something that will be important to them, something that they will identify with, or that they recognize is worth their time in thinking about and trying to think through. The second one is that they need to be working on this same problem. You can’t say teams one and two are working on this, three and four are working on this, five and six are working on this. Third one is that they structure in so that they make a specific choice as the outcome. Because it’s easier to solicit team feedback if everyone is making a specific choice rather than having kind of an open-ended narrative response. And it helps to stimulate whole group discussion as you’re moving. Now it can sound like it’s limiting to say that you have to make a specific choice, but you can do in a very broad way. And the fourth one is simultaneous reporting. So all the teams are asked to report at the same time on what the choice was that they made, so that they can’t piggyback off another team who’s putting in effort. So, as an example, one of the courses that I taught in the psychology major in Texas was the tests and measurements course in psychology, and test and measurements starts with a stats review. They’ve all had statistics, it usually comes prior to tests and measurement. But it’s the first time that they have an opportunity to work with statistics in kind of a decision-making way. So you start with a stats review. So one of the activities that I would do, I gave them two hypothetical first-grade teachers with how many questions 10 of there students got right on a spelling test. And the two distributions had the same mean, but one was fairly normal, and one was highly skewed. So they had to do their quick statistics review… Do the mean, median, mode and standard deviation describe the shape of the distribution. But the question I was asking them was, “If you were the principal, which teacher would you offer an after-school tutoring program to for extra pay? And which teacher would you potentially nominate for a teaching award?” They found that question to be a really interesting question. For one thing, students think a lot about what good teaching is, and what constitutes a good teacher. So they already come in with very strong opinions. And they also understand the complexity of, you know, if everybody’s passing but people aren’t excelling, is that good teaching? Whereas if most people are failing, but a few people are getting an “A” is that good teaching? …and how the data contributes to good decision making, but can also be kind of manipulated to contribute to decision making in not such a good way. So instead of just saying, “Let’s review the stats, here they are,” it was a question with a specific choice that they simultaneously reported on. And then we could discuss together. And of course, their answers are different. There’s different rationales in both ways. So then we could discuss together what their rationale was, if they want to debate they can debate a little. It generates a lot of student enthusiasm, and everybody’s doing it instead of just 10% of the class.

John: And once they’ve committed to an answer, they have a stake in and they really want to know, that’s something we’ve seen a lot of things we’ve talked about in the past, too.

Kristin: Absolutely.

Rebecca: When you were doing the team-based learning, were you sticking specifically to problems that were on a class-by-class basis still, like you were discussing in the co-operative setting, or were you doing some longer term activities that went across multiple class periods?

Kristin: I had the… what I consider gift… to often be teaching in a three-hour time slot, which is my very favorite time slot. So I would have activities that would extend two or three hours, but typically not between classes, I found that to be more of a sweet spot, at least for me. At my previous institution we had a very high commuter population. And I promised, in both models, that I would never ask them to do something out of class with their teams, that was one of my rules… that it was just simply too burdensome for students who have multiple outside of school commitments… family and work, or living potentially 150 miles apart, which was not unheard of. I promised them no out of class stuff. I structured that intentionally so that the individual preparation that they were doing, they could do anywhere on their own time schedule, but they were expected to be there. And their team expected them to be there to be able to engage in class. And it was also one of the ways that you talk people into it, when they say “I worked with other groups who were all slackers and we would always set times and they wouldn’t show up.” And I said “That’s not going to happen in here. We already have a time we’re all going to show up together.”

John: And the philosophy that’s very similar to the flipped classroom approach where you let students do the easy stuff outside and then give them assistance with or have them work in a framework where they’re getting more assistance with the more challenging issues.

Kristin: Absolutely. I think TBL [team-based learning] is definitely a flipped classroom approach.

Rebecca: I think the other thing that helps too with that model… of making sure you’re not working outside of class… really helps students with really different backgrounds start working together, because you might have students who are more traditional who are on campus. And so for them to meet outside of class is often not such a big deal. But then if you have students who are working or have families, and there’s a disconnect in the class, even, between those two populations, that helps make that more obvious and work a little bit better,

Kristin: Right. Absolutely. Yeah. And I didn’t want to set up anything where people were made to feel like unvalued team members, because they couldn’t do what was asked of them because of other commitments. Since that was in my control, I wanted to make sure that people felt welcome.

Rebecca: I’ve tried to even do that with long-term projects. In the field that I’m in, we tend to do things that go across class periods, but there’s always the “Are we going to do this outside of class or are we going to do this inside of class, and I try to have them do anything that needs to be collaborative, and decision making, in class, and then things that can be done on their own, even if that means doing some creative work, or whatever, outside of class. But those are independent things that can be done for the same reasons. And I find that students will try to manipulate that system, so that they’re gonna: “Oh, we’ll just do it outside of class, because we don’t want her to know whether or not we’re on top of something,” or whatever. But I call them out on it, because it’s really devaluing some of that exact thing. People have other commitments and things.

John: You mentioned, you started to use a backwards design approach where you started with the things you want them to remember five years later. Did you have to cut back on the breadth of the coverage in the class, to some extent, by doing that?

Kristin: Yes, I did. When I was going with the straight up cooperative learning approach, I did not have to cut back on the content at all. Without the full redesign, I found I could cover the same amount of material in straight lecture versus in a cooperative setting. But it was all coverage. It was just a different kind of coverage. When I approached it from a backward design perspective, and I really was able to focus on the objectives that I thought were important, I did have to reduce the amount of things that we were covering. I have no regrets about that, of course, because I completely recognize that covering material isn’t just covering it. What are students going to do with something I covered in class? They didn’t cover it, I was the one who was learning it and talking about it. So I’m much happier with an approach in which I am consistently hitting on the objectives that I really want them to recall, and that they are working hard to apply those throughout the semester.

John: If they’re not going to remember it passed the final exam, covering more material isn’t terribly useful.

Kristin: No.

John: We talked about that in a previous podcast with David Voelker, who talking about the coverage approach in History…

Kristin: Right.

John: …which is the same logic.

Kristin: Exactly. And I actually now consider that to be a complete waste of time. So why am I spending class time on something that I actually don’t really care if they remember, it’s not the most important thing to me, and they really don’t care if they remember.

Rebecca: You have some compelling arguments for why team-based learning and collaborative learning are good options. If one wanted to start moving in that direction, what would you suggest their first steps be?

Kristin: For team-based learning, there are a couple of great books that are very easy to approach. There are several great resources for team-based learning. Larry Michaelsen published a book in 2008, for example, that covers that from front to back. It gives examples of applications in different disciplines. There’s also a book published a few years later on team-based learning in the social sciences and humanities. That also covers the basics, but has applications that are more specific to social sciences and humanities. Team-based learning has really caught on in medical education and in business education. So in the original book, there are more application examples that are in MD preparation or in business schools. So if you’re looking for other examples, the second book might be a good choice as well. And that one is edited by Michael Sweet and Larry Michaelsen.

John: And in fact, I read your article, or

Kristin: Oh, did you?

John: …your chapter in there as background.

Kristin: I’m glad someone read it.

John: Now I have to read all the others. But, I, at least, did read that. It was very good. So for faculty who are moving to this, what are some pitfalls that they might run into? Or what sort of problems might they encounter?

Kristin: Team-based learning as being a much more structured approach… Michaelsen does a really nice job of laying out the pieces that he thinks are critical. And I agree they are critical. So, for example, he talks about explaining, and testing the model with students on the first class day, and you cannot skip it. So the very first class day, I give students an example individual application test, like they would get for their readiness assurance. It includes basic psychology knowledge that may or may not be present in the culture. So they have some chance of getting some of them right and some not. And then I have them do it as a team. And the team scores, of course, are always dramatically higher than the individual scores. And the team testing process is so much better. [LAIGHTER] It’s more pleasant and interesting and collaborative than they expect it to be. That simply going through that, it allays many of their fears about what a team is going to be like to work with. Plus, when they see that the team has tripled their individual score, they’re like, “Hey, maybe I could depend on other people to help me learn, and maybe this will pay off for me.” So going through an explanation of what the rationale is, having them experience it a little is really, really critical in helping them stay open minded while they experience it. And then regularly throughout the semester, I will keep reinforcing them with those messages. I’ll say, look at this amazing thing you guys did. You used all the intellectual resources around you, and you analyzed this difficult problem and came up with some great solutions. I’ll remind them how much they’re learning and what kinds of challenging tasks they’re able to do as a team when they have the preparation to do it, which helps as they’re starting to think “Well, wouldn’t it just be easier if I could do this by myself?” It helps them to kind of remember, ”Well, yes, but you wouldn’t be doing this, you would be doing something not as challenging, not as integrative.”

John: and probably not learning quite as much either&hellp;

Kristin: Yes. He also emphasizes an aspect that is also emphasized in cooperative learning… of helping the teams develop and giving them feedback, helping them give each other feedback. That’s also really critical, especially very early in the semester, as they’re starting to develop group norms and bond together to make sure that you don’t short the time in class for them to have some group processing time and to build their team skills. So, for example, when I taught last spring, I had a student who came to me after I think it was the second week. So it’s very early in the semester, and she said, “I really need to reassigned teams. My team hates me, they won’t make eye contact with me.” She was really upset. And I’m reluctant to reassign people teams, because often what they’re experiencing, they take with them. It’s not always a function of that team process. So we talked some, and I tried to get a handle on what she was experiencing. I knew where she sat, I had an idea of the team composition. And I asked her to try one more day, just one more day. And then we would talk about reassigning her teams. And that day, I was sure to build in plenty of time for group processing, where they talked about what they were doing well as a team and something to improve. Their team turned around immediately. She was a relatively assertive person, which I already knew. I knew that she could handle this. So she went back to the team. She was able to talk with her team about not feeling heard. They immediately turned around in the way that they were with her. And by the very next class day, they were a relatively high functioning team. They did well all semester. They brought doughnuts for each other. I mean, it was a really nice supportive group. What they needed was the time in class to do some processing. And if I, as the instructor, had been moving too fast, and not giving them time to do that, and not giving them a prompt to do that, it would have been a really negative experience for her. So, also building in time for the team to develop and prompts for them to do that.

Rebecca: So you mentioned liking to have a three-hour teaching slot.

Kristin: That’s my favorite. It’s not required.

Rebecca: So, in that amount of time, how much time would you designate towards this group processing, for example, to give people an idea of what that proportion or the amount of time to dedicate so that you don’t shortcut it and you don’t rush through it?

Kristin: If I were to do an activity that might take an hour, I might spend 10 minutes for group process, it doesn’t have to be very long, or even five. And you don’t have to do it every time, you could do 10 minutes after the first one or two more intensive activities, and then not do it for another few times… and another five minutes just every so often to help them resolve their underlying dissatisfactions and to recognize that what they’re doing is not just application activity, it’s also group interaction. So please take time to do both. Another really important required component that I didn’t mention is peer evaluation, I always incorporate peer evaluation as part of the grade.

John: How did you form the teams in these classes?

Kristin: They’re heterogeneous, first, with a very open process so students can see it happening and know there are no shenanigans… that this is all very open… talking about the rationale that people of different backgrounds bring different strengths. So you want a group that has people of different backgrounds, so you can have a larger kind of learning base between you. So usually, I’ll pick a few characteristics that might be important in that kind of background. And I will line them up around the room based on those characteristics. And if it’s 200 people, it’s a really long line. And then we count off. So when I teach introductory psychology, students who have had a high school psychology class usually are starting a big leg up on the other students. So I’ll include that as a characteristic. Sometimes I’ll include the distance that people are coming from, because then they have different experiences, depending on what class I might also include if their student athletes, just because if you put too many together in a team, then they’re all gone on the same day. They have interesting backgrounds, but they also have patterns of attendance and of absence that need to be adjusted around. And we’ll count off all the way around so people can see how the teams are made. But heterogeneous teams are really, really critical. Having students with pre-existing relationships will throw off the team process in a way that automatically excludes people that don’t have pre-existing relationships… plus they tend to be lower performing teams. And I don’t want to set that up on purpose.

John: One of my colleagues once did this in a class of, I think it was about 350 students, but he just sorted them alphabetically. So he had them organize himself that way, and it was a fairly long process. But, it was kind of amusing for those of us wandering by and just seeing…

Kristin: …this huge line… Yeah.

John: He didn’t do it that way In the future, he used other criteria.

Kristin: I’ve had colleagues that I’ve talked with that think that this is a long process. It’s not. You can sort 200 people in 10 minutes, and then you’re done for the whole semester,

John: Doing it alphabetically…

Kristin: takes a lot longer.

John: …can be more challenging, because they were self forming that… it didn’t convert rapidly.

Kristin: The other thing I never do is I don’t put the students who didn’t come the first day into a team, because there are characteristics about why they didn’t come the first day. If you put them all together in one team, they share some of those characteristics… It tends not to be a very high performing team. So I make sure they’re sorted out among the other teams. But that was one of the things that I learned in cooperative learning. That, before I did cooperative learning training, and I was assigning group work, I would assign people based on if you didn’t come the day we did the assignments, you were in another group. And that group typically did not do very well. And as an instructor, it’s my responsibility to create a learning environment in which students can excel, it’s on them whether they do their part. But if I’m setting up a team in ignorance, with predictable characteristics, so that they’re going to have a failure experience, that’s on me to correct. And it’s not on them. So afterwards, I felt guilty when I had come to a new realization. But, yeah, it’s my responsibility to set up an environment in which those students can be successful in their teams.

John: In your chapter in that book, you mentioned that when you switched over, it did affect your course evaluations a little bit. Can you tell us a little bit about that?

Kristin: Just a little bit. But yes, it did. So when I was doing straight lecture, I was shooting for engaged lecture. And in psychology, you can build in little experiences, especially in introductory psychology, where the topics are changing frequently, you can always build things in that are kind of interesting. You can do a little optical illusion here and a little bit of memory trick there. And there’s these ways to build it in, but it is still basically straight lecture. And I got high evaluations for that. I was careful about trying to build those in every day, you know, every few minutes. And when I went to cooperative learning, where it was essentially the same approach, but in in a much more engaged and cooperative fashion, those evaluations stayed very high. Students knew each other, they were happy in class. When I went to team-based learning and I was actually asking every student to participate all the time, and be prepared in class in a way that their contributions were much more obvious than mine. My evaluations did drop just a little bit, not a lot, but a little. And I am grateful that I was teaching in a context where I knew that my department wouldn’t care. They were more interested that I was doing good teaching. And they understood the many factors that influence student evaluations. But I also recognize that it’s incumbent on me to help students understand how they are learning, what kinds of things encourage learning and retention, and then you kind of let the student evaluations fall where they may.

John: When I read that, it reminded me of that study that came out a few weeks ago from Harvard in their physics program, where they found that students in active learning classes did demonstrably better on tests, but they perceived their learning as being lower. So there was a pretty strong inverse relationship between their perception of learning and actual learning. That seems to be fairly common, there have been a number of other studies where what students think to be most effective, is often not what most enhances their learning.

Kristin: Right.

John: Do you have any other advice for our listeners, who might think about using either collaborative or team-based learning in their classes,

Kristin: The one thing I would say is that teaching a cooperative learning or a team-based learning structure class is a lot more fun. You have to be willing to give up control, because when you’re lecturing, you have absolute control… meaning even that students can’t ask you weird things, because you haven’t opened the door for that to happen. But when you structure the learning experience, and then you give up the control to the students, it is an exciting environment to be in. I wasn’t as tired when I was coming out of class. I was energized, you could feel the difference in the room just walking into class… they were excited and talking with each other. When I would circulate around before class started, they’re talking about the class instead of talking about other stuff. It completely changes the environment in the classroom in a way that I think really matches what I expect out of a university education for students, it creates a environment of intellectual enthusiasm around the topic that you’re teaching.

Rebecca: So we always wrap up by asking what’s next?

Kristin: That’s a great question. So right now I’m 100% administrative. And since I’m in a new position, in a new institution, I’m gonna spend some time figuring out all the newness pieces. But I’d like to go back to the classroom, at least for a course here and there when I can. There’s nothing different about students than there is about people. So I also think often about how what we do in the classroom, what we understand works and what we understand doesn’t work, how that applies in administrative settings as well. We know for example, that people tend to try and find the shortest path. So if they’re trying to learn something, they want to put in the least effort to learn it. If you ask a faculty member to do a task for the department, they are obviously going to choose the easiest path to do that… not necessarily the best path. So how do I take the experiences of learning and teaching, that in some ways are better understood to an environment of administration that in some ways is not as well understood? What kinds of lessons can I apply there as well?

Rebecca: Well, thank you so much. It’s been a really interesting conversation. I’m sure it gives a lot of people things to think about as they move forward in this semester and future semesters.

Kristin: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

98. Developing Metacognition

Many students arrive in our classes with relatively little understanding of how they learn. In this episode, Dr. Judith Boettcher joins us to discuss how well structured project-based or problem-based learning activities can help students develop their metacognitive skills so that they become more successful as learners.

Dr. Boettcher is the author of many books and articles on higher education and has long been a leader in the field of online education. The Online Teaching Survival Guide: Simple and Practical Pedagogical Tips, co-authored by Judith has been an important resource for faculty transitioning to online teaching. At Oswego (and many other institutions), many faculty have been using materials that Judith has developed for ACUE (the Association of College and University Educators).

Show Notes

Transcript

John: Many students arrive in our classes with relatively little understanding of how they learn. In this episode, we examine how well structured project-based or problem-based learning activities can help students develop their metacognitive skills so that they become more successful as learners.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Today our guest is Dr. Judith Boettcher. She is the author of many books and articles on higher education and has long been a leader in the field of online education. The Online Teaching Survival Guide: Simple and Practical Pedagogical Tips, co-authored by Judith has been an important resource for faculty transitioning to online teaching. At Oswego many of our faculty have been using materials that Judith has developed for ACUE (the Association of College and University Educators). Welcome Judith.

Judith: Thank you very much, Rebecca and John, it’s great to be here.

John: We’re really pleased to have you here. Our teas today are… are you drinking tea?

Judith: I actually… yes, made a special point. Here’s my cup, which you can’t see. But I chose one of my favorite teas, which is a lemon and ginger tea from England, of course,

John: the Twinings version.

Judith: No, this is a Hamptons tea from London.

John: We have the Twinings version of that in our tea collection.

Rebecca: I’m drinking Jasmine green tea today.

John: And I’m drinking oolong tea today.

Judith: Sounds good.

Rebecca: Judith, your mug looks really interesting. Is it abstract art. Is that what was on it?

Judith: It’s actually pears.

Rebecca: Okay, I only saw the bottom part of it. I can see it now.

Judith: Yeah, right. Well, I will confess that at some point, I finally decided to clear every vendor cup out of my cupboard.

Rebecca: That sounds refreshing.

Judith: Yes.

John: I have vendor cups all over… in my vehicles… in my offices, everywhere.

Judith: Well, that was part of my retirement process that I went through. I said, “Okay, that’s it.”

John: One of the things we had trouble with is picking a topic because you’ve worked on so many topics, but we settled on having you talk a little bit about how students can work to improve their metacognition using project-based or problem-based learning. But before we do that, could you talk a little bit about what metacognition means?

Judith: I would really like to John, partially because I had this book with the title of metacognition that I was reading when I was with a family event. And one of my relatives said, “What in the world is this? Meta what?” [LAUGHTER] So, an easy way I like to think about metacognition is the definition of just it’s thinking about thinking. It’s a definition, I think. that we can all just really grab on to and we can really use. But then I kind of like to expand that definition into one that’s in the How People Learn report that I go back to pretty regularly and that is that “metacognition is the process of reflecting on and directing our own learning.” And I really like that one, because it’s got the two steps, I think, what we want to really kind of focus on with metacognition, and that is reflection… really stopping thinking, pausing… and then actually directing our own thinking, because that leads to action. So then we have reflection and action, which I think is the core of metacognition skills.

Rebecca: One of the things I think we often talk about in education context is this reflection piece. And we always tell students to reflect, but we don’t always give them the time and space to do that.

Judith: Yeah. [LAUGHTER] Very much.

Rebecca: So can you talk a little bit about project-based and problem-based learning and how metacognition connects to those rather than standard ways of operating in the classroom?

Judith: Yes, I would love to do that. In fact, that was one of the first things when we’ve got into online learning, was that there was a real struggle as to how do we maintain the security and everything of people taking tests. And so it turned out that we decided that one of the best ways of gathering evidence of student learning was not by doing these tests… that we would actually have the students do projects. And that kind of evolved into the following process. And that is that in some of the work I’ve done for ACUE, and also in the book, I’ve mentioned that I really like to design a course that starts with the students selecting and doing a project, actually in week one or two of the course. And then that students actually focus on that project throughout the entire course. And that’s the mechanism by which we gather evidence of student learning. It kind of also avoids this whole process of buying papers and buying other kinds of things. Because you really have milestones along the way.

I’m going to stop and talk about the first step for just a moment. And that is choosing a project. One of the wonderful things about choosing a project is that then students actually have to stop and then think about the kind of project that they want to do. It gives them an opportunity to actually customize a course to their interests, which starts getting past that big motivation problem that sometimes teachers might say, “Well, how do I motivate them?” Well, we give them the opportunity to choose something that is of interest to them. So they choose a project, and then they actually write up what that project is that they want to do. But that’s not the end of selecting the project. Before the project is really kind of finalized that they’re going to be working on, they actually then sit down and talk with some of their other students and the other peers. They switch and swap their proposed project descriptions, so that they actually talk out loud about the project. And then hopefully, by talking to their peers about it, they get some additional ideas, and they refine it a bit. And then it goes to the faculty member. So the faculty member doesn’t get it right away, but it goes through this first the individual students thinking about it, and then the other students thinking about it, and then the faculty member can take a look at it. And that’s only milestone one that can take up to about the third week of a course. But by then, hopefully, that gives them a real focus of the course rather than having just the topics thrown at them from week one through week 16… that they’ve got a real focus of: “Oh, how is this going to affect my project?” We can come back to that I just want to mention briefly then there’s like four other milestones, five other milestones for every project throughout the course. And the first is that project description, then the second one would be planning how do I plan to do it, which is a really important metacognitive skill. And then another milestone would be some type of other “Just checking in, how are you doing?” …kind of a thing. And then there’s two final things. One is where they actually share their project with the other students, like in a mini conference, whatever. And then the final thing is the actual final thing, which might not be a paper, it might be a video or an interview, it could be one of a number of things that would go into their student portfolio then. But, that gives the student a focus throughout the entire course that way. That’s a lot, isn’t it?

Rebecca: It is a lot. [LAUGHTER]

John: When you have them give feedback to each other, do you recommend that that’s done synchronously or asynchronously If it’s an online course?

Judith: You know it’s actually best, I think, that the students do it somewhat like a brainstorming event. What you and I are doing right now here with Zoom, students can do with FaceTime, they can do it with just an online gathering chat, whatever. If they’re really, really busy online students, and they have to do it asynchronously, that can be done too with email. I mean, that works too. But it’s often really great for the students themselves to talk out loud. We don’t have them talking enough, I think. They read passively, and they kind of think and everything else, but we don’t have ways for them to use their voice to talk about what they’re thinking.

John: It sounds like the project’s really nicely scaffoldied. But how do you bring in the metacognitive development? How do you get students to improve their metacognitive skills? Is that something they’re explicitly thinking about? Or is it something that’s done as part of the structure of the project?

Judith: Well, of course, it depends on how the faculty member wants to do it. As I had the chance to go back and look more intensely at the metacognitive skills, it occurs to me that thinking metacognitively is such a basic intellectual skill in many respects, particularly now in our 21st century. It’s as fundamental as reading and writing. And, you know, I think we need to look at our entire curriculum from pre-K through whatever as to really how do we explicitly teach and model and coach metacognition skills, which includes with that initial project proposal, we can talk about the fact that what they need to do is think and be sure to build on their interest to give them the criteria and coaching as to how that’s going to really work for them. It’s really a problem when they have to select a project. So what are the constraints on that problem? What are the features or benefits that they’re going to get from this? …and to build that into the assignment and actually writing out the initial proposal and then meeting with their colleagues and peer students, that’s all part of the coaching and the modeling of the metacognitive process.

Rebecca: I think your emphasis on talking about things is really interesting. Because I find that students often will passively write something, and it makes no sense to anybody. But as soon as they try to explain it out loud, they realize it makes no sense. Because when it’s in writing, or they’re just reading in their head, they don’t often have that realization. But as soon as you try to say it out loud as a sentence, it’s not structured in a way that makes sense. They catch themselves or they realize, “Wait, there’s a big hole in what I’m talking about here… nobody knows what I’m talking about.” You can tell it by other people’s faces looking blankly back at you.

Judith: And then they realize that they don’t know what they’re talking about.

Rebecca: Yeah. [LAUGHTER]

Judith: So yes, in fact, that’s apart of the power of talking with their peers about the project, because the peers would most likely say, “Well, wait a minute, why did you choose this? Why are you interested in this?” And so that’s when they have to dig more deeply. It almost goes back to that Socratic questioning, “Why do you think that? Why is that important to you? Do you think this is going to make any difference in your life?” One of the things I really recommend in online learning is that the students, when they read and look at the objectives for a course, to stop… and in the first week, actually, to have the students process those learning objectives, and then personalize one or two of them and say, “This is what I really want to make certain I know when I finish this course. This is how I’m going to be a different person.” Thinking “What skill am I going to be able to do when I finish this course?” They kind of set that goal and set those expectations early on. Hopefully, students setting those goals and objectives become again, more personal to them, and something that they can check themselves. And then as they go through the course, “What progress am I making on this particular goal? How is this coming?” …kind of fun.

Rebecca: Are there ways that you would suggest a faculty member model metacognition early on in an online course?

Judith: Well, I think a really easy way for faculty to model meta cognitive skills is… say, for example, in biology or one of sciences, one of the things I love to look at is the biography of a scientist and looking at, “Well, how did they come to this point? What made them think X, Y, or Z as opposed to A, B, or C?” and we find it’s really rooted in their personal lives and their thinking that they’ve been doing. So faculty member can somewhat do the same thing, in fact, very early on in their introduction to the course. Now, what do you teach Rebecca?

Rebecca: I teach web design.

Judith: You teach web design, okay. So in your introduction, then, to your students, you can say the reason I love web design is as follows. And you can go back into your life and your experiences and say, This is what happened to me… and this is what I was thinking… this is how I got to this point. So you really speak out loud, and you share your processes by which you arrived at that point. And we can do that for any kind of thinking… as we’re talking about an experiment. We do want faculty to share their expertise and their dissertation, for example, they say, “Wow, you know, this is why I’m interested in this. And you know, after 25-30 years, I’m still interested in this because of this.” So they share their thinking processes. Another really easy time is when students ask them a question. And obviously, this can happen online in the synchronous activities. And the students might ask them a question about, “Well, gee, Professor, so and so what does this really mean in this instance, or in this context?” And the faculty member can say, “You know, I really am not sure about that. And I would like to, before I answer that, I would like to do a little more research and thinking, and I will get back to you on that.” So it sends the message that the process of thinking is something that we keep doing all the time.

Rebecca: I think that moment of admitting that you’re not always the expert in everything in the moment, is always a good thing for students. And they respond really positively to that.

Judith: That’s right. It also gives them the opportunity to say, “You know, Professor, I would like a little more time to think about that. Let me get back to you on that.”

John: How can we tell that students have improved their metacognitive skills? I like the idea of having the peer instruction do that. Or to work with students to help them recognize what they know and what they don’t know. But how can we measure that? Or how can students know that they’re more metacognitively aware? How can they observe improvement?

Judith: Well, one of the things I did as I was thinking about this was, you know, I was kind of preparing for you asking me the question, “What are the actual metacognitive skills?” and I came up with five thinking skills that we really want to encourage and model for students. The first one is an easy one. And that is, number one, we really want to encourage students to think. I know that sounds somewhat simplistic, but oftentimes think about in the online environment, we give them an assignment and students, they don’t even read it, really, they kind of scan it. And they kind of assume without thinking that that’s what that assignment is going to give back. And so one metacognitive skill is to pause and stop to really take some time to think and process. In fact, I kind of like to think about the assignment as a briefing. You know, we use briefings in business and politics and detective crime solving and all the rest of it we use briefings. Maybe as a faculty member, you want to encourage the students to think of an assignment as a briefing and a briefing is, “Well, this is what we know now, what do we want to know about next? How are we going to find out what that next is? And when you finish this assignment, what do you expect to know or think about or become clear about?” So step number one is really pausing and thinking. There’s a book out recently called, was it Kahneman? It’s that Thinking Fast and Slow or Slow and Fast?

John: Yes, Thinking Fast and Slow. It’s a great book.

Judith: Thinking Fast and Slow, right. Well, you know, he makes the point that we live in such a fast-paced world that our first response to anything is that, “Oh, I know the answer to that.” It’s like Jeopardy, and you really don’t stop and pause and think. And so, because it’s really easy, that’s the short-term memory, we don’t have to think about it. We know that in order for learning to occur, that we really have to stop and give time for information going into our short-term memory to get into the long-term memory. The only way anything goes from the short-term to the long-term memory is for us to connect what we know already with the new information coming in. And you know what? …that takes time. So that’s the one thing for all of us. Time is so seriously, we don’t want to take the time. But if we don’t take the time, the only way learning occurs, you know, if we grow dendrites in our brain, and so if we don’t take the time the dendrites don’t grow, and nothing lasts. It’s kind of a fast bullet shot, so to speak. And then we forget it. As soon as we use it in a sentence, it’s gone. So the idea that we have to really discipline ourselves to stop and think and process, what is it that I’m needing to do? What is it that I want to learn? What do I want to get out of this? And when I finish this assignment, how will I know that I’ve really finished the assignment. Which brings us to the next thinking skills. One of the things I really liked, I use myself a great deal. And this is kind of a hard won practice of my own. And that is that, whenever I was given an assignment whether it was at work or wherever, whenever I had a hard time getting started… you know, getting started is sometimes one of the most difficult parts of anything, right? So anyhow, I finally figured out that one of the reasons it was hard getting started was that I didn’t quite know what it was I was going to do. I finally realized that my best practice is that I visualize what it is going to be when I finish. I get requests to review articles for journals, etc, etc. Well, I know now that my visualization works is that I’ll be done when I finish answering all those questions, you know, read the article, finish answering all the questions, I compose my response back to the editor. And that’s when I’m done, okay? So with any kind of a project, we want to encourage the students to say, “Okay, what is my assignment going to look like when I’m really done? When I have finished reading this article, or reading this core seminal research project? When will I really be done? Will I be done when I can explain the research to someone else? Will I be done when I can actually implement these ideas in a web design project? Is that when I’m going to be done? Just what is that project going to look like when I’m really done?” So we just celebrated the Apollo landing on the moon, in July. I just had the recent opportunity to see the Apollo 11 program that they created out of the original footage from everything that was happening at Cape Kennedy and Houston… the pictures, it showed like, literally, a room full… it almost look like 100 guys, and they were all guys at that time… 100 guys sitting at computer terminals, and you think when they worked on that project, how did they envision success? They had to envision success as actually a man being on the moon. How did he get to the moon? What did he land on the moon in? And then how are we going to get him back from the moon safely? I mean, think of all the things that had to be planned and worked on and everything had to be coordinated to make that thing happen.

So thinking skills, we have to think, we have to visualize, and then we have to plan. Once we know what that final vision is, we need to then plan each of those steps along the way. And again, we can model some of that and coach that by building the planning into the assignments for the project. We definitely want the students to give me a plan with the date, the milestones, the resources they’re going to be needing,]… if they need to make appointments or interviews when that’s going to happen. So we help them realize that projects just don’t happen. They happen after all of these various steps. And then, of course, we build in step number four, evaluating and pausing to then debrief each step along the way: How am I doing? Do I need more time? What else might I need? What else do I really want to know, if I do have enough time? I mean, this really does happen with web design, right? We get to a certain point, we think of it’d be great if I did this. Great if I did that. Do I have enough time to do that? Do I have the knowledge to do that? Or do I need to learn a new skill to insert that into that. So those are all questions that we want to ask encourage the students to plan for and to ask along the way. And then of course, the final one is the final debriefing. When you hand something in, you get feedback both from your peers and from the faculty member. And in many online courses, you do have these little mini conferences where you invite alumni or experts or just friends in to say take a look at this and see what you think.

Rebecca: I like your framework of the briefing. And it’s actually one that I use in my classes pretty regularly.

Judith: Oh, great.

Rebecca: When I’m doing long term projects, I have students on a weekly basis do basically a little briefing of what did they do? What do they need to do? What are the planning steps? So not just like that big scope of the whole project, but on a very routine basis, checking in with their schedule and checking in with their plan and what they’re struggling with to demonstrate what they’re doing, and what their thought processes is as they’re working on it. So I asked them, “What were the big design decisions you made this week? And what did you base those on?”

Judith: Wonderful. In fact, I sometimes like to use the example of Mark Harmon, the actor in NCIS. I love those programs, actually. Like every morning, he kind of just drives into the office and says, “Okay, what do we have.” And each one of his team members have to then report “Okay, since we saw you yesterday, this is what we’ve done. And this is what we know.” And then from that briefing, decide on their next steps, “Okay, we need to do X, Y, or Z.” It’s a really kind of a nice example of planning.

John: Building it into the project just seems to make an awful lot of sense. If you want students to improve their skills, have them apply those skills and structuring it so that they’re doing it is a very reasonable way of doing this.

Judith: And I love the fact that, Rebecca, as you were saying is to ask the students to say what have you been thinking for? And what made you arrive at the decision that you’d like to do something a little different there? So yeah, again, verbalizing the thought processes, is part of the metacognitive abilities.

Rebecca: I think one of the things that surprises students with that assignment is that they don’t realize that they’ve made decisions.

Judith: Oh, interesting.

Rebecca: That happens in projects. But also, if they’re writing a paper, whatever, they’ve made decisions, but they don’t necessarily think of it like they’ve made a decision. They’ve just kind of moved forward. So it causes them to stop and pause and do that first thing that you were talking about, and think for a second before they move forward.

Judith: Yeah.

Rebecca: I do have to say, that since I instituted that, my students are far more articulate when they’re talking about their projects. [LAUGHTER]

Judith: When they’re doing their own projects, they get their passions involved. There was another thing I wrote recently on curiosity, how important curiosity is and how we want to really build that in. In fact, one of the things when I was doing a workshop, I suggested to faculty, I said, “You know, let’s get the students to stop answering questions. Let’s get them posing questions. And let’s give them problems that, in our discipline, we don’t know the answers to, because what fun is that? Ok, so we set up the situation that we the faculty know the answers, and then they have to figure it out. So switch it around a little bit and say, here are some problems and we don’t know the answers, how am I to approach this? And then we’ll give them the challenge.” And that really, particularly John, I’m sure you’ve seen this with the TIP program, the students come up with things that you never would have expected.

John: It’s one of the reasons I enjoy it so much.

Judith: Yeah.

Rebecca: So one of the things that I think can be a challenge is if you’re doing a big project throughout the semester, we don’t want it to end up being just one big high-stakes assignment. Are there methods or ways or strategies that we can manage bigger assignments to have some lower-stakes moments or ways so that there’s not so much pressure, and that they’re allowed to make mistakes and improve and learn?

Judith: Yes, and I’m glad you asked that. Actually, somewhere in a couple of tips online that I wanted to refer to. It’s, I think, tip… is it 38 and 60, that talk about project-based learning online… that’s something to think about. But I also have a chart that recommends the grading process for an online course. And so you assign points to each of the milestones in the project. So you assign points with the project proposal and selection, but you also then have smaller team-based events where it’s worth a little bit. Also, the discussion board is a really important aspect of online learning. But it’s hard. How do you evaluate and grade that? So I recommend that about 15%, even 20% on the discussion board. And it’s pretty much a given like the classroom discussion is… so long as students are reasonably consistent about following the rubric for the discussion boards. So you’ve got a little bit there, you have also shorter essays and shorter leadership opportunities in an online course where they might summarize a week’s discussion. So there’s other kinds of activities within an online course. At no point is everything dependent on that one big project, but it does require the students to invest time and energy in that.

John: Many people often think of a need to develop metacognitive skills most for our less able students, thinking of the Dunning-Kruger effect, and so forth… that those who know the least often overestimate their learning by the most. But I think those issues may apply for students at all levels.

Judith: Yeah, we assume that great kids, actually, automatically have these skills and think that way, and they don’t.

Rebecca: Which I think is also a really great thing to talk about, too, because we often talk about metacognition, and helping students who are struggling, do better. But it’s also a great way to challenge students who are already doing well, to do better.

Judith: Absolutely, a great point.

John: For the last 32 summers, I’ve been teaching in the Talent Identification Program (or TIP) at Duke University. And one of the things I’ve observed is that many of the students there have generally been able to breeze through their regular classes without ever having to really learn how to learn. So they sometimes face a little bit of a challenge when they arrive at TIP, and suddenly they’re faced with a challenge. But a really nice part about this is that it’s an ungraded program, so that they can develop their learning skills without worrying about what sort of grades they’re going to get in their classes. And it’s much better to do it there than it would be at some future point, perhaps in a physical chemistry class, or a differential equations class, or some other class later in the career, when they haven’t really had to develop those metacognitive skills that are going to be useful in their future. Developing metacognitive skills really are important for students all along the spectrum.

Judith: That’s a really good example too, John, it’s an interesting phenomenon to talk about, really.

John: How can we tell whether students have higher levels of metacognition? Or how can we tell when students have not developed their metacognitive skills?

Judith: Well, you know, this is obviously a great question. And so let me share just a little bit about my thoughts on the students. How do we know when the students are totally clueless about their thinking processes? And I think one really red flag, particularly in online courses, that these are the students who rather consistently post comments on the discussion board: “What are we doing now? What was the assignment about? When is this? Oh, there’s a rubric, I didn’t know that.” These comments often are the ones from students that they really haven’t taken the time to really read the assignment and to make plans on that. These are also the students that we know for online students that they’re trying to do too many things. They’ve got 1000 things going on with their families, and work, very likely. And they just simply don’t take the time. So I think… just simple reminders all the way along the way in an online course from the faculty member. In the assignment, to be really clear about the assignment and to say, “Be sure to think about the following.” For example, if an article has been assigned, that you as a faculty member say, “Okay, this is why I’ve selected this article. Here are some of the core concepts that are really important in this” and guide the reading to say, “Why do you think the person did this? Be sure to be able to answer these kinds of questions?” And then also to be clear about maybe what you want to do when you finish this reading assignment, to talk about it. If necessary, explain it to your 12-year old. See if you can explain it to that person. So being really clear in the assignment would be, I think, super helpful for all students in an online course, because again, it’s the kind of thing a faculty member would do in the classroom, you’d say, “Okay, I want you to read this article. And you know, this is one of my favorite articles. This is why I want you to do it” …and all the rest of it.” I think we need to do more of that in the assignments in the online course. That’s one thing.

The other students that are clueless are the ones that keep thinking that all they have to do is passively read and reread and reread. We hear that all the time from students that “Oh, well, I read it 15 times. I don’t know why I don’t understand it.” Without saying, while giving help and modeling, as you’re reading this, think about the following.

Just as an aside, I signed up to take a course on brain, dendrites, and synapses from the Hebrew University of Jerusalem, and oh, my gosh, lesson three, they said, “Well, you know, if you’re not familiar with the electrical circuits, you probably need to go into the Wikipedia and get all this information.” I’m thinking, “Oh, I’m lost right here.” But the idea was that there was some foundational content that I didn’t have. And so again, if in an online course, if some of their content is dependent on some of those other core concepts that they might not have, to remind students that, “hey, this is a difficult reading. And you may want to get some further help in X, Y, or Z.” So again, going back to that whole briefing, and using the assignment as a briefing and giving them clues how to do well in the reading. Some of those comments really have to do with whether students are struggling or not. Because we want to emphasize more clearly the reasons for why we make choices. When we design a course, we really make lots and lots of design decisions and selections as to why I want my students to read X, Y, or Z. And I think sharing that rationale and sharing those reasons, I think, really helps give insight into the metacognitive thinking of the faculty member as well.

Then John, you ask the question, where we talked about little bit about how even when students are doing well, they may not be thinking metacognitively. And I think that it’s important to recognize that these explicit thinking skills about thinking, about visualizing, and about planning, etc, that it would be a good idea to build those visualization and planning into the assignments.

Now, one of the challenges with online courses, and I’ve seen this for years, is the fact that some students don’t have good places to study, they can be living in very busy environments, they don’t have an office, they don’t have a really quiet place to go. And so one of the things that we can do is actually ask the students to post a picture of where they’re going to be doing their studying. It kind of gets them thinking, “Okay, where am I going to be doing my studying? Is it going to be a place that I can really concentrate?” There was one study hint… that I actually just pulled it out, because I gave it my granddaughters…. and it was called “How to study” …and the little hint in there was that when you are sitting down to study, what you do is you find, select or design, some kind of a flamboyant little hat… think Cat in the Hat kind of thing, you know… just flamboyant things, design some kind of a flamboyant hat so that when you put that hat on, you tell your brain and you tell the other people around you that you’re going internal now… you’re really going to think… you’re going to work, you’re going to study. We almost do that nowadays… when I go into my Starbucks, or my local coffee shop… that’s where I do my writing, by the way… I put on my Bose headphones, because again, that signals to the people around me that, “Hey, I’m working, I’m really concentrating.” So if we asked the students to really stop and think, “where am I going to study?” If I’m a family person, and I’m working, “When am I going to study?” That I have to schedule my week, and the times and places that I’m going to do my work and do my studying. And oh, by the way, I kind of have to get my family and my friends on board to say “Yes, John or Rebecca, they’re studying now I can tell they’re studying, you got to leave them alone, give them the time and the space to leave them alone.” For online students I think this is super important for them to build a schedule ahead of time. And again, it recognizes the fact that metacognitively takes time and it takes space in order to do it well.

John: That reminds me of a couple of our previous podcasts, a few episodes back Mathew Ouelett from Cornell, when we were recording, he mentioned that he has a drawing or a painting, I think he did of a tomato, I don’t remember if it was a painting or if he colored it, but he has a big poster with a tomato on his door that he puts up as a signal for faculty to leave him alone because he’s engaged in a pomodoro technique, and he wants to be just focused on this. In a much earlier podcasts, our very second one, Judie LittleJohn was here and we talked about a metacognitive cafe online discussion forum she and I had both used. And one of the things she uses in it, and Rebecca has introduced that in her classes too, is exactly what you suggested, having students describe their study space and perhaps post a picture of it. And then to address all those issues about how well it works, how they deal with distractions, and so on. This ties in nicely to some of our earlier discussions,

Judith: Well great, actually as I was in my Lucky Goat local coffee shop, actually getting ready for this podcast a couple weeks ago, it turns out this young business person came in and he had a briefcase and all the rest of it and I could tell he was kind of settling in… a little too close to me… but that’s okay, there weren’t many seats available. But anyhow, he was talking pretty loudly. And I thought, “Oh, he’s on his phone.” You know, there’s everybody who talks to themselves these days is on the phone, right? And anyhow, we ended up talking to each other partially because he was talking. He said, you know, what he was doing was he was talking to himself about what he was going to do during his time there. And obviously setting his own personal goal. And then he had learned that from his mother. [LAUGHTER] But that does remind me of something else I do want to share. And that is when I do sit down… my own metacognitive practice… when I sit down to do a task… say, I go to the coffee shop, and I got a couple of things I need to do, I write out a mini plan. And list the time I’m starting… list my first subtask, the second sub tasks with approximate time to completion, and everything else. And of course, it doesn’t happen exactly like that. But I do get done, and I do manage to check the pieces off. But then another really important metacognitive practice that I’d like to share is that when I finish that task, I say “What is my next important step?” In fact, David Allen and his book Getting Things Done on Stress Free productivity, thee whole environment, even in business. His primary question is, “What is your next step?” So before you stop where you are, particularly in a writing project, you write down what your next step is. Because it’s totally short circuits, that transition. Because when you sit down, “Yeah, what was I doing now? What do I have to do? What is my next step?” Hey, I can’t tell you the number of hours that that particular little hint has saved me in terms of really making progress and stuff. So that’s another little hint to build into all that project planning. “What is my next step? What do I have to have next?” And sometimes for online learners, particularly, it’s not sitting down and studying, they have to get a resource, or they have to get a book or they have to make an appointment or they have to do something. But then that’s something that doesn’t have to necessarily be done exactly in order. To kind of almost wanting to build a little calligraphy thing, saying “What is my next step?” is a really, really great little hint.

Rebecca: I think experience certainly teaches us that, but it’s something that students who have less experience don’t explicitly know to do. Because we all know that we’ve tried to cut corners and hurry through something. And then if we don’t do that… I know I’ve spent hours figuring out now what was I doing? Like it’s been a while since I’ve worked on this project. Now, where was that? What was my filing system?

Judith: Yes. And what was I thinking? And oh, dear, I wish I bought that along with this other thing with me. Yeah, exactly.

Rebecca: I also wanted to circle back a little bit to the workspaces too. One thing that I found with asking students to talk about their workspace is also to show them professional workspaces. And to talk about the different kinds of environments and how space can help facilitate certain kinds of activities, but also can short circuit certain kinds of activities as well.

Judith: I think that’s really, really helpful. I know, I can’t tell you the number of conversations I had with people when I was in a work environment, and management was making all kinds of decisions that were not conducive to good collaborative work or good independent work. I mean, it’s a real discipline to think about.

John: And you need both spaces, but you will often only have something that’s better for one or the other.

Judith: Exactly, exactly. For any kind of a course, as we’re talking about metacognition, we get overly focused on “what” the students are learning as opposed to the “how.” So just in a capsule comment, that as we are designing online our classroom based courses that we really include in our assignments.. in the design and the various activities… that we include both the “what” of the content and the “how…” How do I get there? And I think we started getting there when they start setting those personal goals. Because some of the courses now are starting to include a goal or a learning outcome that they want their students to think like a scientist. You’re not just learning biology, we want you to think like a scientist. Or we want you to think like an engineer, or an entrepreneur, or whatever. But getting this… it’s a different mindset. You just don’t want to learn what biology is and learn about the content of biology, but I love finding out about the biographies of like Einstein and a few other folks. It’s just fascinating to get into their heads as to the process that they use to do that. Eric Kandle, by the way, is another Nobel Prize winner. He’s got some fabulous book out about his thinking and his processes by which he actually investigated and learned about memory. So… fun.

Rebecca: I think that using biographies is a great way to introduce students to that a little bit.

Judith: Yes.

John: A number of software packages, such as Lumen Learning’s Waymaker package, Norton’s Inquiszitive, and some of McGraw-Hill’s and CENGAGE’s products include attempts at building student metacognition in their products. For example, they’ll ask students questions and they’ll also ask them about their confidence in their responses. The Norton Inquizitive package, in particular, sets it up in a somewhat game-like situation, where they get to bet points on how confident they are. And then it gives them feedback on how they did versus how they perceived they were doing. Do you think this type of approach might be useful?

Judith: I was just reading a research study about that, John, and the students… Who was that by? I would have to go back and find it, I’ll have to email it to you. But the results showed that the students who were less confident…

John: …did better.

Judith: Yeah. Who was that?

John: I don’t remember. I think it was on the POD list, but maybe not, it might have been on an economics list. Consistent with the Dunning-Kruger effect, the students who did relatively poorly, had relatively high self perceptions of how well they did that were not reflected in their test scores.

Judith: …So important to be asking the question: “What do I know?” and “What don’t I know?” Because that’s a core, isn’t it? If we can answer that question. In fact, one hint that’s in that book called Make It Stick… which is a real good one… one of the key things I took away from that was the technique or practice of having in a textbook, taking the heading and turning it into a question and then seeing if you can answer that when you finish reading something. So again, “What do I know when I start? What do I know when I finish? And am I able to answer that question? Or am I able to pose another good question based on that?” So explaining to ourselves what I know and what I don’t know, I wanted to go back to that study that we were just both talking about, because my question that occurred to me as I was reading this is that, you know, the problem, I think, is that the study really didn’t really give the students an opportunity to verbalize why they felt confident or less confident, which is, I think, a whole missing piece of that. And I don’t know how they obviously would make designing a study… actually, my dissertation, in fact, I will bore you with the title of it is “Fluent Readers’ Strategies for Assigning Meaning to Unknown Words in Context.” And the thing is that, half the time, they didn’t know that was an unknown word, they just assumed they knew the word. So again, if we don’t know we don’t know something, we can be very confident. [LAUGHTER] But anyway, in order to get the answers to that question of the study, I actually had the students talk out loud to me, they verbalized and I would go back and ask them about the word and say, “What were you thinking when you came to that?” …and all the rest of it. So they had to verbalize their thinking. For me, it was a good study, it really worked. So again, going back to the value of verbalizing,”What do I know?” and “What don’t I know?” and “What do I think I know?” And Rebecca, we got that when you said, “Well, you started talking about what to do, and then you realized you didn’t know anything.”

John: As you suggested before, a really good thing with any project is to think about where you’re going next. And we always end our podcast with the question, “What are you doing next?”

Judith: What a great question. Okay, what am I working on next? I did mention that I just finished a little video for the Distance Learning Conference in Madison in early August. On my back burner, and I’ve been saying this for a couple of years, I would really like to write a book on concepts. What I’d like to do is… I mentioned I love working with faculty, and when we design courses it so often on a topic-to-topic basis. And yet, as I started working with faculty towards the last 10 years or so, I started asking faculty to tell me what their core concepts of a course is. And, you know, faculty, that’ve been teaching for 10,15, 20, even 30 years, they would pause and they’d say, “Uhhhh, [LAUGHTER] you know, we just teach topics,…” as opposed to core concepts. And we think about core concepts are what stay with us when we finish a course, hopefully. So the question really is, what do you expect your students to know, five years, 10 years down the road from what you have been spending all this time and energy on? So I’d like to write a book about thinking about concepts and how to design a course, and focus on problems and concepts rather than topics.

Rebecca: I like that, I think that would be really helpful. Can’t wait to read it.

Judith: [LAUGHTER] I can’t either.

Rebecca: You gotta visualize, you gotta visualize. [LAUGHTER]

Judith: Thank you. You can give it right back to me.

John: We talk a lot about backwards design. But a lot of the classes that many of us teach were not designed in that way. They did not start with those major course learning objectives, and then work backwards to get to that point. And they’re just series of topics taught in the same way that they were taught to them when they were students. And they were taught in the same way as their previous generation taught them. And there’s not always a lot of thought going into that. And that sounds like a really good project.

Judith: Well, thank you, I may call you and see. I’ll need faculty to work with on that project. So I may contact you for that.

Rebecca: I’ll sign up.

John: Be happy to.

Judith: I’d be willing, God be willing that I get going on that. [LAUGHTER]

John: I think we all have a few projects like that. But eventually they often happen.

Rebecca: Eventually.

Judith: Yes, well, actually talking about it is a good thing. Because the more things we write down, and the more things we actually talk about are more likely to happen.

Rebecca: And I have just told the world so it’s gonna have to happen, right?

Judith: Oh, dear. [LAUGHTER]

John: Although if you change your mind, we can edit it out.

Judith: Scratch that…

John: I know I come up with some things after a podcast where I say I want to do this in my class next semester. And once it’s in a recording now, I pretty much have to do it. [LAUGHTER]

Judith: Well, John, before we break up here, when you do go to Duke, what do you teach?

John: I teach economics, introductory micro and macro economics.

Judith: Okay. Sounds great.

John: It’s been a lot of fun. I love doing it. The kids are just so amazing.

Judith: Well, kids are amazing at that age. They really are. It’s wonderful to see them evolving to young men and women. You know, I’ve got eight grandchildren. My oldest is now 19 and a half. In fact, she did microeconomics online, both one and two this summer.

John: Thank you for joining us.

Judith: Well, thank you very much. I really enjoyed being here.

Rebecca: This is a lot of fun. Thank you so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

94. Open Reflection

Students can provide useful feedback on instructional practices and class design when they are asked, In this episode, three students from John’s spring economics capstone class join us to provide their reflections on the class’s experiment in developing an open pedagogy project. Our guests in today’s episode are Maria Aldrich, Victoria Heist and Charlie Tararzona.

Show Notes

Transcript

Rebecca: Students can provide useful feedback on instructional practices and class design when they are asked. In this episode, students join us to provide an open reflection on one class’s experiment in developing an open pedagogy project.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guests today are Maria Aldrich, Victoria Heist and Charlie Tarazona, three students who participated in the creation of an open pedagogy project in one of my economics classes this spring semester. Welcome.

Victoria: Thanks for having us.

Maria: Thank you.

Charlie: Yep, excited to be here.

Rebecca: Today’s teas are…

John: None of you are drinking tea, are you?

Maria: No.

Victoria: No tea.

Charlie: No tea today.

Rebecca: How regretful. [LAUGHTER]

John: It happens with many of our guests. I’m having ginger peach black tea.

Rebecca: And I’m drinking Lady Grey. The issue is our tea selection is no longer close to our recording studio. It’s a problem. It’s an epidemic now with our tea choices.

John: …now that we’re recording in this little closet in a building next door, where at least we don’t have toilets flushing every 30 seconds or so that we have to edit out.

Rebecca: So John, can you start first by explaining what open pedagogy is, to kind of frame our discussion?

John: Going back a step further. Last year, I saw a presentation by Robin DeRosa who presented on this at the SUNY Conference on Instruction and Technology and she made a really compelling case for some of the advantages that open pedagogy projects have. And open pedagogy just involves having students create content that is open and shared publicly with the world.

Rebecca: So what class did you do an open pedagogy project in?

John: This class is a capstone course in the economics department here. It’s taken mostly by seniors and a few juniors. And it’s a seminar course in economic theory and policy. It’s one of our smaller classes. And we had only 27 students in this capstone, this semester.

Rebecca: So why this class?

John: Because the students were ready for it. The course builds on all the courses that they’ve had up to this point and it allows them to pull together material they’ve learned in all of their prior courses, as well as the cognate classes in statistics and math and so on.

Rebecca: So what kind of project exactly did you propose to these students?

John: I originally proposed two options. One was to do something on behavioral economics, because past classes have found that to be a lot of fun, and another one I suggested was they could just pick current topics and work in small groups and create papers on that. Turned out that they really didn’t like any of those ideas and given the nature of open pedagogy, I left it up to the class to decide what their topics would be. And I think it was actually Charlie, who came up with the idea. And would you like to tell us what that was?

Charlie: I know you had mentioned in the beginning of the class the idea of open pedagogy. And I found that pretty interesting because it seemed like a good opportunity for us as seniors and juniors to really put what we had learned out there. And also, in terms of topic selection, you gave us the opportunity to really choose which topics we wanted to talk about. We ended up choosing the topic of intergenerational mobility and economic inequality. We focused more on the economic inequality aspect of it in the end. But yeah, like I said, it was just a good opportunity for everybody to really finish their college careers with something that they can show.

Rebecca: Dr. Kane is going to close his ears now and you guys are going to tell us exactly what you thought when he said, “Hey, you’re going to write a book.”

Victoria: I was hesitant at first, just because group projects are kind of daunting, especially in economics. However, a collaborative group project was exciting to do… to see all of our work put together. As economic students it isn’t something you really see, it’s usually individual work.

Maria: Oh, yeah, I would agree with Victoria. I was kind of hesitant at first, especially because it was something new for our class so I figured there are probably a lot of kinks that needed to be fixed. So I was a little worried about not having everything fully figured out at first. I thought it was something interesting. It appealed to me because I like the thought that other people could read what we had written and we could have control of what we would want to talk about.

Charlie: And the topic and the idea of a book project really intrigued me… that it just let us put out there what we had learned over these past few years and gave us something that we can show in the end of it.

Rebecca: Were any of you scared?

Victoria: Not scared. I wouldn’t use that word.

John: Were you concerned?

Victoria: A bit concerned, just because I like doing my individual work. I feel stronger in that.

Maria: Yeah, I would agree. I think group work can sometimes be difficult to have for every class because everyone has a different writing style and everyone works on their projects at different times. So I think at first, you’re a little bit worried that not everyone will be able to work well together. But I found that in my group, we were able to work very well and we’re able to meet once a week to go over what we needed to work on for the week.

Charlie: Yeah, I found something similar to that experience. Whereas my group, after the first few weeks, figured out what we wanted to do, and when we could meet, and what was the most effective way for us to put the book project together? And I think it turned out really well.

Rebecca: So you’ve all mentioned groups, can you talk a little bit about what the groups were, how they were determined, and how that worked?

Victoria: The groups were groups of three from the class, because there’s 27 people. And then we’re able to email Professor Kane and ask if we wanted to work with anyone specific, like if we had friends in the class, we could work with them. But if not, or if we didn’t want to work with someone we knew, we’re able to randomize it.

Maria: I was put into a group of two other random people that I didn’t know, but we were able to set up a group chat immediately and communicate very well through that.

Charlie: I actually emailed Professor Kane about being a group in Victoria and we also included another student in that. I think it worked out pretty well and I was happy with how it turned out.

John: Before the groups were formed, though, the class decided on what the topics would be. So we had kind of a free-form planning session where we narrowed it down to nine topics you wanted to address. And then at that point, we knew how big the groups were going to be. And it worked out nicely with three people per group.

Rebecca: How did each group get assigned a topic?

Charlie: So the way we assigned topics was, we had created a list of the nine topics, and then each individual group could choose their top three, and then we divided them that way based on everybody’s top choice. If they didn’t happen to get their top choice, they usually got their second or third, I think that only happened for maybe two groups, and they seem to be fine with what they ended up with.

John: And going back a little bit further, it was a weighted voting scheme that you didn’t just rank them… that you assigned points, if I remember was it 10 points I gave you? And so if you really wanted to chapter you could bid all 10 points on that. And if you were indifferent, you could have assigned weight to your top three preferences and so forth. And it did work out really nicely where I think most groups got their top choice, but two of them ended up with their second or third choice, but it seemed to work.

Rebecca: How did you find collaborating in the end?

Charlie: I found that it worked really well meeting every week. We also had presentations every week that we gave on specific topics that we’re talking about during that week. So that set the initial schedule for us to meet every week and talk about what we were doing and what was going on. Also with the book project at the time, in terms of organization, I found it very laid out and simple.

Rebecca: That sounds like you had a writing group that met that frequently, but it also would be more of like a study group as well?

Charlie: Yeah, I would definitely say it was a mix between a writing group… a study group. Your group members ended up being the way, if you wanted to succeed in the class, like that was the way to do it was to work cohesively with your group members.

John: And it should be noted that they had other tasks in the class as well, where they selected topics that were presented each week and each group was responsible for presenting an article or a research paper on a topic, some of which were related to the book and others were completely different. The groups were persistent across all the assignments and involved more than just writing the book.

Rebecca: How’d you get feedback to make sure whatever you’re putting out in the public was good enough?

Maria: Well, we mainly used Google Docs. At first, we tried to use hypothesis. But that wasn’t really working out well. So we ended up just going back to Google Docs. And each group would be given a couple of chapters to review each week. And they would write a couple comments in that chapter as well as some comments made by our professor and we used that and we also used each other’s feedback to make those edits.

John: How did that work? Where the comments helpful?

Victoria: No. [LAUGHTER] Just because I’m very protective over my work, which I know I should be open to criticisms. However, I got some comments sometimes I was just like questioning, like instead of “what about this article that you might want to look at” it would be “change this word,” where I know we emphasized that often in class many, many times, but still people would persistently do that.

John: I hope that wasn’t from me. Was it from me?

Victoria: Oh, no. Well, if you wanted to do that, you’re the professor. You can do that. But you also give us feedback that’s helpful. Whereas, students I feel like if they’re rushed or doing it, like 20 minutes before the class, they’re not going to look at me like, “Oh, what about this topic that might be interesting to consider.” Instead, they’re like, “Switch this word.” That just might be the students in the class too, just because we did have a lot of work in the class. And I found that because our group would meet to practice our presentation before the class, a lot of groups are doing their final work 15 minutes before the class.

John: Yes, the quality of the work did vary a little bit across the groups and across the individuals within the groups. Overall, there was some really high quality work, and all three of you did really well. But the quality of the feedback varied quite a bit.

Rebecca: So the feedback was generally done outside of class? Like not during class time?

John: Primarily, except for the presentations on the work where there was some feedback during the presentations.

Victoria: Yes, but I found that your feedback was most helpful, rather than the students.

Maria: Yeah, I would say I paid a lot more attention to Professor Kane’s feedback than some of my fellow students. Luckily, we didn’t have that experience. We had a lot of people give a sincere, really constructive feedback, but sometimes I’d be hesitant to take that feedback because I didn’t know that was the direction that I should be going in. But I definitely think our experience was a little better and our comments were more substantial, I would say.

Victoria: And I think next time maybe switching the groups that review the feedback might be helpful, because if you have one group that gives worse feedback, and you keep getting that, it’s not as helpful.

John: The way it was structured was each group reviewed and provided comments on three other groups, and we did that on three stages. And the class decided to maintain persistent groups there. I did give them the option, but I think it does make much more sense to vary it so you’re getting a wider range of feedback.

Charlie: I think the idea to keep persistent groups stem from the fact that we wanted to have somebody read the paper and then continue to read the paper throughout the weeks when we were supposed to be improving it or making it better. So then they could also see the changes we were making. And I agree with my classmates where I think we can say that it didn’t work out too well. There’s some groups just didn’t happen to give feedback that was too good.

Victoria: I also think part of it was the length of the papers because each of us had to review three full papers for the weeks that we did that, and three 20-page papers is a lot of reading to do on student written economics. And I think maybe in the beginning it was helpful to read all three, but maybe as time went on to scale that back a little bit, so we don’t get burnt out.

John: More detailed feedback on a smaller number of papers.

Victoria: Yeah because at first, I find myself doing it too. Like the first paper, I’ll take the time to read every single word and provide helpful feedback. But I can see myself not doing as much on the third.

John: I gave feedback in three different ways. The first time I gave video feedback, and while I’ve heard that that can be really efficient, I was taking about two hours or so per paper. And that was really slow and people really didn’t like the feedback that much because some of the feedback was fairly long in terms of the suggestions. So, I probably gave a little too much feedback. The second was with comments embedded in Adobe. And the third time I just basically went along with everyone else and provided the feedback directly in Google Docs. And the nice thing about that is I was able to see some other suggestions and sometimes I’d say, “Well, maybe that’s not such a good idea.” Because in many cases, the original draft actually made more sense than the feedback.

Rebecca: What way did you all like feedback better?

Charlie: I think the best feedback I received was actually in class feedback when I would go to Professor Kane and ask him, “Hey, you know, this is what’s going on with my paper. Is there something else I could look at? Is there another source I can find?” I found that to be the most effective in helping me write the paper. I was also a fan of the comments in Google Docs, they were pretty helpful.

Maria: Yeah, I think the most helpful feedback were the comments from Google Docs because, for that last draft, I was able to go through and resolve any comments that I had made the changes to and that just helped motivate me to make my draft a really good copy. And then I would say that I was really against the video feedback because I personally like to review feedback multiple times. I like to go through it and make changes to it. And I found myself just typing up his comments at the end of the doc so that other teammates could see it as well. So I was definitely against the video feedback and prefer the Google Doc comments.

Victoria: Yeah, I prefer Google Docs as well, just because I could see where exactly you wanted the changes done. It gave more specific feedback and then it also gave the students validity I guess, like this should be changed. Like I made a comment in one paper and said, “I think you mean a different word.” And they just resolved it and moved on. But then Professor Kane came through and said, “Yes, I agree.” And I think that you agreed, and they now are aware that yes, those changes need to be made.

Rebecca: I think sometimes when a faculty member responds to student comments in a way that it also helps students know how to make better comments. So it would be interesting to see how another round of that would have gone after Dr. Kane had responded to some of those right? To see if the comments were better the next time around.

John: Yeah, I think I should have done that from the beginning. And I’m sorry, I didn’t. But in the future, I’ll probably use Hypothesis. Now that we have Hypothesis in Blackboard it will be much easier. Among the problems we had is that people had some trouble making comments on Google Docs because they also had edit access to those and they couldn’t mark up specific text. And with PDFs, that was a bit of a problem given the way the browsers were set up that they had to change a program in order to make comments on PDF documents. So now that we have that in our learning management system, it’s going to be much easier to do that and the comments will be a little more persistent, because one of the issues was people were, as you mentioned, resolving comments sometimes before anyone else had a chance to see them. And the strategy was to have the draft documents with the comments copied over to another folder, and they were only supposed to make changes in their working document, not in the documents used for comments. But there were three or four people who through three drafts, just didn’t quite get that notion and I’d see the email saying that comments were resolved, and I would go back in and unresolve them. But in any case, there were some problems with those. That’s an issue that I think has to be worked out a little bit more efficiently.

Rebecca: Beside some of the technical issues that we mentioned, what were some of the biggest challenges of working on a project like this?

Charlie: I think one of the bigger challenges was keeping the cohesive idea behind the whole book where the topic we had chose was income inequality and we also had talked about intergenerational mobility. But as the book progressed, we kind of saw that portion of the book fall off a little bit where chapters were really focusing on the income and economic inequality topic.

Rebecca: So is that something you discussed in class to keep everybody on track?

Charlie: I think we mentioned it at one point towards the end, we’re just like, “Okay, are we going to keep this? Are we going to not keep this?” And I think we agreed, we could talk about it but we won’t make it a major portion of the book.

John: There was also some scaffolding on the project… that it didn’t just start with people starting to write, groups were first asked to put together a bibliography, and then an annotated bibliography, and then an outline of the chapter, and then the actual writing started after they had feedback on each of those steps.

Maria: I would agree with Charlie, I was definitely worried about the cohesiveness of the entire book. But for my group, specifically, we did a very broad topic, the global trends of economic inequality, and for myself, it was really hard to find relevant subjects to talk about because it was just such a broad topic. It was really hard for each of us to find something that we could spend a large amount of time writing about. So I’m not sure how the other groups felt. But for us, it was definitely hard narrowing down what we specifically wanted to talk about, and then to find resources that were recent enough to include.

Victoria: Yeah, I agree with you on that. I think one change I would make after we figure out the specific topics, you can go deeper in that because it’s hard as a group to form a thesis statement or very cohesive argument because we ended up doing more of a timeline than like an argumentative paper because you have to split it up.

John: Your topic specifically was on what?

Victoria: Tax-structure and income inequality. So basically, we looked at early 20th century, later 20th century, and the 21st century, and how the changing tax structures led to increasing income inequality over time. So that’s kind of how we split it up. But I think if I was to do it again, I would take a different approach to it, because I did the first section and finding information on World War One income inequality is much harder than it seems. So I struggled a lot with that too.

Charlie: Yeah, in terms of how we wrote our chapter of the book, I’m usually a fan of writing papers that follow a timeline as an explanation but that’s just a personal preference. It doesn’t work for everybody. So I can definitely see how making the cohesive argument along with following that timeline can be pretty difficult.

John: In your chapter, I think the timeline made a bit of sense. We were talking about the evolution of it and the transitions in your chapter were pretty smooth. I don’t think that worked as well in all the chapters, quite often it looked like they were three essays…

Victoria: Yes.

John: …chopped and pasted together.

Victoria: There was this one paper with a bunch of sub topics, but it wasn’t cohesive. And I was reading it and it just did not make any sense to me how it was organized. So that was one of the suggestions I made… maybe taking a step further in class and presenting maybe our papers a little earlier.

John: In more stages…

VICTORIA. I was just trying to read it and I just could not make sense of the organization of it, where maybe if we caught that earlier we maybe could have made better paper.

John: I was giving them feedback in several groups… that sort of feedback… that they need to smooth out the transitions and have a more logical structure. But some groups responded really well and did a nice job with that, other groups were a little more reluctant to do that.

Rebecca: Perhaps some groups will respond really well to some peer pressure. [LAUGHTER]

John: And having the presentations in class would have helped do that. When people in the class were saying, “This is just too disorganized.” And most of them got better by the end, but it was a stretch getting there.

Rebecca: So you’ve talked a little bit about some of the challenges, but what was really rewarding about working on this project?

Victoria: I found it really helpful to work with the group. I had Charlie and then another student, Junweii, in my group and we all read each other’s parts. I know I went through the document and made comments for my own group too. And we were all able to bring it together, make comments for each other, ask each other questions about like what sources to use especially too. And it was easier in that regard than an individual paper. Because if you make a mistake and you don’t realize it, no one’s there to help you, it’s just you. But here we have people to help each other.

Charlie: Yeah, I always find it beneficial to complete a task with other students also trying to complete the same task as you. It just makes the learning more interesting. And you’re more willing to go and spend the extra hour looking at the document to just make sure you understand what you’re writing, but also that it fits with whomever else you’re working with. I found it really beneficial or satisfying just the fact that we, like I said, could create something that any ordinary person could probably read and understand what was happening.

Maria: Yeah, I think the most rewarding part for me was just seeing that finished product and getting you know, positive feedback from Professor Kane and from my other group members. I think working in that group setting helps to motivate me to do the best of my ability. And I think it was just rewarding at the end to see everything come together well.

Victoria: I think it gave us all a deeper understanding of the material too because, instead of writing it yourself… because you can write something and not understand it. I’ve done it many times. [LAUGHTER] But when you’re sitting in a group, getting a presentation ready, you each need to understand the material. So you’re explaining what you learn to each other. And that’s something you don’t get by yourself. I found that really rewarding.

John: What about the public nature of the project? The fact that this will be out there, it will have your names on it, and it could be out there indefinitely.

Charlie: I found that portion of the project pretty intriguing and exciting. Just like I said, you can go out there, and obviously we’re college students, we’re looking for employment after this. So just showing an employer, “Hey, I’ve written something that’s been published. It’s out there, you can go read it for yourself and see what you think.” It gives something for the students to show.

Victoria: Yeah, it made it exciting because we knew what was at the end of the project. Rather than just a finished paper, we actually had something to like prove ourselves, like we did this.

Maria: I think for me, it was cool to know because at the end of the semester, I’m able to go to my family and say, “Oh, here you go. This is something that I worked on all semester long. Here’s something that you can read and you can better understand what I’ve studied for the past four years.” So I think it was helpful that I was able to show my family I’ve worked hard on this. This is something that is to show for that.

Charlie: I would definitely concur with that. Economics as a topic isn’t really discussed when you’re talking just with family members, so many of them don’t understand what you’re talking about. And you’ll try, but it’s hard sometimes. So to put something together that they’d be able to read and understand, I found that pretty satisfying.

John: And how did the class select the audience for this? What level was it written for?

Victoria: Students with a background in economics I think we decided on. But we came together as a class and decided on that. But you need economic background to understand some of the things we wrote.

John: But at an introductory level, so it wasn’t written at an advanced level. It was written for people who’ve had an economics course somewhere along the way.

Victoria: Or just no background. You don’t have to go to college to read the book.

Rebecca: How would you change this project in the future? We touched on a couple of things here and there, but do you have any other key things that, if the same exact project were presented to another group of students, how would you change the structure? Or the way it’s organized? Or the way that it’s presented the first day?

Victoria: Thinking about the class as a whole rather than just the book project, we did weekly presentations which was a lot of work in itself. So I would probably minimize those and focus on the book. Because we were sitting there reading 20+ page economic journals every week and making a presentation on it and doing the book project. So I think having more time dedicated to the book project and presenting on that material, rather than just economic journals that people have written, like it gives background, which is helpful, but maybe a little less, or maybe shorter ones, or ones that are just easier to understand. Because I know a lot of times you would say, like, “I know you guys don’t understand this, it’s challenging. But we still need to know it.” Like you would explain it in class, which would be helpful, but reading something you don’t understand is really difficult for students… in economics specifically. That’s challenging.

Maria: Yeah, I would agree. I think, at the beginning of the semester, it was a lot of work to have to juggle both the presentations and the book at the same time. So I kind of like the idea, I’m not sure if it was you Victoria, who mentioned it in class, of doing the presentation one week and then the next week working on the book and having class time devoted to the book in the week after. I think that would have been very helpful too because we did meet as groups, but if we were able to meet in a class setting than I think other classmates will be able to make comments on your chapter and offer advice. I think it would just help overall with the workload that we have.

Charlie: I also agree with that. I think the improvement can be made where we’d work on maybe a random topic every other week, and do a presentation on that, and then also incorporate the book project into that. I think it would help with the cohesiveness of the book along with just feedback and all the other problems that we had discussed.

John: One of the things I had suggested at the very beginning, you may recall, is I suggested one option is to spend the whole class focused on this. Another option is just to do it the way it was done in the past, or something else. And the class actually voted for the something else. Now having had the experience, the something else didn’t work quite as well, and that more class time should have been devoted, I think, to this and I saw that too.

Victoria: I think we’re just looking for something exciting. Like yeah, it’s a book project like we know what we’re going to do with that. But the presentations just added something else, but if I went back to a book project because then we could have taken the steps at a slower pace too, like the annotated bibliography, like the topics, we could have taken way more time with that than we did. Because once we did that very quickly, and then went into presentations, and then we just had due dates instead of meetings in class.

Maria: Yeah, I think for us, what appealed to us with this combination of the book and the presentations was that the presentations offered structure for us when we knew what we were getting with those presentations. We knew each week that we’d come in with the presentation. And I think with the book, we were excited because it was something new and different and I think we were a little too hesitant to go fully and choose the book, because we weren’t sure what we would be doing in class. We weren’t sure how we would be tested on that. So I think the combination of fields lost because we were able to have that structure, but we were also able to try something new.

Charlie: I know for some of the students in the class they had mentioned to me… they were hesitant to get rid of the presentations weekly because they were a fan of learning something new every week and learning a different topic, not just focusing on the book project. They really wanted to increase their knowledge base by just learning about multiple fields of economics. So I think that’s why we ended up going with what we went with in the end. But I think we all could all agree that if we had done that every other week, it would have been more efficient.

John: I agree. And I think some combination might be good for the reason you mentioned, but more class time devoted to it would be helpful.

Victoria: Maybe at first too, do a presentation. Like the first presentation, I don’t know what week that was, but maybe keep that one because when our group really met each other, we worked together, and then we planned a time every week where we would meet.

John: And if this is done again, and that will be if the class wants to do this in the future, perhaps that first topic for the readings could be related to whatever they choose to do so they’re actually doing some scaffolding with the presentations then.

Rebecca: I had something similar in my classes before where a team formed early on. We did something small, low stakes, to figure out how to work with each other and what doesn’t go well. So that when we did something a little more high stakes, you already knew what the wrinkles were going to be so that you could plan for that moving on. So it sounds like your presentations served that purpose, whether or not you intended that to happen or not.

John: But it became a lot of work when it was done every week, in addition to writing a book.

Victoria: That was difficult.

Maria: Yeah, I think it just helped to make us all more comfortable with each other and more comfortable speaking in front of the class.

Rebecca: So the big question is, of course, should other faculty do this?

Victoria: Yes, I’m working on my honors thesis right now, which is kind of what you would do in a traditional seminar. And it’s very difficult. So just having people there… write it with you… know what you’re talking about… You can ask them questions. In our group chat, we often ask, “What would you recommend for this part of the paper? Or what articles do you think are appropriate for this?” If you’re doing it by yourself, it’s very difficult. And the overarching topic… I feel like in a lot of seminars, they have that. It’s a topic for the seminar, but it doesn’t really filter through as well as the book project does, because we are all cohesive, all of us together working as a class of 27 people, which you never see. So, I found it really helpful and I liked it a lot. And it wasn’t like a crazy amount of work. You did the work, and you study, you did the presentations, and you wrote a paper, but it didn’t take you hours every day to work on. I feel like I learned more in this class than I have in other classes that I write individual papers for.

Maria: Well, I think I would partially agree and partially disagree with that. I think as a class, we all appreciated that Professor Kane was willing to change like the class structure and was willing to try something new. And I think that was definitely intriguing for us and provided something different as our last economic course. But I think if I had done my own topic paper, I think I probably would have learned a little bit more, I think just I would preferred that. But I think it was still important to get this experience and try something new.

Charlie: I think I would definitely suggest it to some other faculty members to maybe try this out. Like Victoria was saying, working with a group is pretty beneficial. And I feel like, from a personal standpoint, I learn more when I’m working with other people who I can ask questions to, get feedback from. Really, it helps your understanding of the class. In terms of incentive, I find that I wanted to work on the book project because you had that end goal of, “This is something that I can put out there and show to somebody.”

Victoria: Yeah, but at the same time, group work can sometimes be the worst thing that ever happens to you. Like we got really, really lucky because I know Charlie, we’re friends so we were like, “Okay, let’s work together. We’ll just get one random person.” Junwei was like such a blessing. We just work together so beautifully, but I feel like if we had someone that didn’t want to do the work… wasn’t willing to put in the work… didn’t show up to meetings… that would ruin the project for us. So I don’t know how you could fix that. But just if there’s a good group, it works. If there isn’t, I feel like it wouldn’t work as well.

Rebecca: So good to write one book during your time here, but maybe not many books. [LAUGHTER]

John: But there could be other things. For example, they could have been podcasts that were created. They could be collections of essays.They could be video projects that are put together by groups. So there’s a lot of different things that could be done.

VICTORIAL: Yeah, I would throw that out there. If you did this again with another book, like, yeah, you can write a book, but you can also do that… a different kind of form of the same kind of structure. That would be interesting.

Maria: I would be interested in doing some type of podcast because I know some of my friends in their classes have been required to do podcasts. And I feel like you have to prepare really well for that. So I think maybe that would have forced your teammates, if they weren’t doing the work, to do the work so that they wouldn’t get to the studio and not have anything to say. So I think that would have been another really cool option.

Charlie: I think it would be a good option for capstone classes, just because I know for a lot of majors, you hear what the capstone is about for the three years before you even get there. And I know personally for me, I’m also trying to get a political science degree, my capstone is next semester, like I’m already dreading the 25-page paper I’m gonna have to write. So to switch it up and have the students maybe not know exactly what they’re in for, I think it gives a little bit of an intrigue and like, “Okay, this isn’t just the I’m going to go and write a paper all year. It’s something else that I’m going to do.”

Victoria: Yeah, it’s more fun. I’m more willing to write a paper that my group members are in. Like we can all see each other too in the Google doc and talk to each other in the chat… be like, “What do you think about this part?” Or like Charlie can watch me while I’m writing my part of the paper and say, “This is good. Maybe change this. Or bring this sentence up.” You don’t do that in individual papers and even if you write an individual paper and have peer feedback, it’s not the same as having it right there, real time, or just people caring more because it’s theirs too.

John: We did have some issues with that early on though, in the first draft or two, because there were some people who really didn’t want to try using Google docs for writing. And were any of you involved in that?

Charlie: So, I’m not opposed to Google Docs. [LAUGHTER] I had just always used Word documents before. So it took a little bit of getting used to but once you commit to it, it’s a really nice thing to have in your repertoire. Google Docs, I feel like, is used by countless numbers of people, companies, places, businesses, the college. So honestly, as a student, you should just take the incentive to try to get to learn it. And once you learn it, it’s really beneficial to you.

John: One of the problems was that some people were writing in Word and then uploading it to the drive and that made it really hard for other people to edit. And eventually everyone switched over, but it did take a few iterations with some people.

Maria: So yeah, I think there were a couple of challenges with having different drafts because people made comments on separate drafts. So I think just sending out a reminder email would be helpful and letting people know because I know I think I made my changes on the wrong draft the first time and we had to send an email right away to have him fix that. So I think just having it set up all before the due dates like before you mention it in class would be really helpful too.

John: Yeah, there were some rough spots. This was new for me too.

Rebecca: So we always wrap up our podcast by asking what’s next for each of you?

Charlie: This December, I’m looking to graduate from Oswego, which is exciting for me. And after that, I’m not really sure what’s going to go on. We’ll see.

Maria: Well, I’m graduating this Saturday, and I’m going to be moving down to Florida for a little bit and doing an internship there.

Victoria: I’m also graduating Saturday and I’ll be working at HSBC this July in their graduate development program.

Rebecca: Sounds like exciting futures for each of you.

John: What are you doing in Florida? An internship where?

Maria: I’m doing the college program, the Disney College Program.

John: Oh, wonderful. Maybe I’ll see you there at the OLC conference. Well, thank you. It’s been a pleasure working with you all semester. And thank you for joining us.

Charlie: Thank you for having us.

Victoria: Thank you.

Maria: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

51. Engaged scholarship

Many of us live and work in communities where there is a strong town and gown divide. Building trust, engaging authentically, and developing deep understanding through intergroup dialogue takes time, patience and the right structure. In this episode, Dr. Khuram Hussain, an Associate Professor of Education and Interim Dean at Hobart College, joins us to explore a model of engaged scholarship that challenges the academy to engage in dialogue with and work alongside the community to address pressing local issues.

Show Notes

Transcript

John:Many of us live and work in communities where there is a strong town and gown divide. Building trust, engaging authentically, and developing deep understanding through intergroup dialogue takes time, patience and the right structure. In this episode, we’ll explore a model of engaged scholarship that challenges the academy to engage in dialogue with and work alongside the community to address pressing local issues.

[Music]

John:Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John:…and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[Music]

Rebecca: Our guest today is Dr. Khuram Hussain, an Associate Professor of Education and Interim Dean at Hobart College. Welcome, Khuram.

John:Welcome.

Khuram: Thank you for having me.

John:Our teas today are:

Khuram: I’m actually drinking coffee. I hope that’s ok.

Rebecca: You and most other people. [LAUGHTER] We’ll let it go.

Khuram: I will end the day with tea.

Rebecca: Ok, perfect. I think we had a recent guest who also ended the day with tea. Today I have chai.

John:And I have pineapple ginger green tea.

Rebecca: Well, that sounds good. You always are far more adventurous than me.

Khuram: If it’s any consolation, I have a little cardamom in my coffee, which I typically put in my tea, but I really like it in coffee as well.

Rebecca: That sounds good. I should try that.

Khuram: I highly recommend it.

Rebecca: Do you have an advice about how much?

Khuram: One. One is good.

Rebecca: One is good. [LAUGHTER].

Khuram: If you want it a little stronger you can crack it and then let it sit and it’ll be even more cardamom(y). [LAUGHTER]

Rebecca: Perfect. [LAUGHTER]

John:We see you’ve done some work with engaged scholarship and service learning. Could you tell us a little bit about what is meant by engaged scholarship for those who aren’t familiar with it?

Khuram: Engaged scholarship is essentially the integration of community needs with learning and it involves addressing community needs along with whatever respective disciplines and skills a scholar may apply to a particular condition. It could be anything from developing a literacy program that is also being useful and utilized in a community, but drawing from that community in order to make sense of what questions you want to answer. So, you’re not drawing it just from a review of literature or from a body of scholarship that emerges from conferences or a community of scholars, but in fact from a variety of voices within the community itself. It’s a much more community relevant approach to even designing research before you actually do it, and it spills out into community engaged teaching as well.

Rebecca: What got you involved in engaged scholarship?

Khuram: I first had the opportunity to do engaged scholarship as a professor of education at Hobart William Smith. I was teaching a course on the civil rights movement and a colleague approached me about volunteering to serve as a professor at a maximum-security prison, and the program there was run by a Bard Prison Initiative where long term inmates were given the opportunity to enroll in an undergraduate program. And so I taught the exact same course that I was teaching on campus within the educational space that they had created for prisoners (maximum security prison) and that was my first chance to think about the ways in which the needs and realities of communities outside of campus and inform the work in learning on campus and could also inform my notions of scholarship.

John:Your work is a form of service learning in terms of the student involvement in it. How does your approach differ from the more traditional service learning approaches?

Khuram: I think that a lot of what I have seen in traditional or conventional service-learning approaches is that there’s a great focus on the ways in which our students will learn by “doing for” communities. So how can we help children learn how to read? How can we provide food to food-scarce areas? And that becomes such a central narrative and the assumptions that young people have about what service-learning is is that we’re gonna learn through service for, and what I think is unique and special about the kind of work that many folks are doing today and I hope to be a part of that (and I hope I have been a part of that) is to do service with. To move from that model means we are required to collaborate and to take a much more team-based approach to service work and the learning then moves both ways. The service then moves both ways, and that I think is the fundamental difference between what we’ve been trying to do the last few years and what we’ve often seen provided to students.

Rebecca: How does your engaged scholarship relate to the service-learning projects and things that you do with students?

Khuram: In part, the ways in which engaged scholarship works is by providing students and faculty and community members an opportunity to create knowledge out of the questions and concerns that emerge in community related work. So for instance, we started an initiative known as “Tools for Social Change” some years ago, and before we looked at any kind of service project we looked at the ways in which the community saw itself. How did long-term residents see college campus residents? How did college campus residents in the same city see long-term residents of the city? And put them into intentional dialogue, first through interpersonal relationship building and then talking about social and structural issues that have informed their understanding of themselves within the city. And within larger structures of identity, race and class particularly. After they developed that understanding we asked, “Ok, what does this community mean to you? Where do you feel empowered? Where do you feel isolated?” Based on the answers to that, we were able to map out a different kind of geography. Even though we had developed a sense of connection and collectivity as members of a community that had been dialoguing all semester, we were operating within a city that was deeply segregated and divided, and so it was from there that we looked at scholarship. We looked at research that we could pursue, and one of the first things that became really important for us to consider was the way in which the economics of the city and the capacity of some to gain access to jobs opportunity was very different than it was for others. And so we ended up taking that initial group and developing wider groups that would go out into the city and inquire… essentially do a self-study of the city about the economics and economic opportunities that were available. And so essentially it was these two stages: first of engaging in dialogue; coming to an understanding of what shared community work could be and then going out into the city with the same participants and essentially conducting appreciative inquiry and having students and faculty and community members (long-term community members) interviewing members of the community, and we were out at the Salvation Army, we were in barbershops, we were in laundromat, we were in every corner of the city and particularly in corners of the city that didn’t often have a strong voice or were not well represented, I should say, in conversations about economic development. We were able to take those, transcribe them and give them to members of the working group that are trained qualitative researchers. They synthesized that, summarized it, and we were able to present it to the city. So, here we’ve created knowledge and we’ve created it through a certain kind of process, right? You might want to call it bottom-up, but I like to see it as horizontal; it’s relational knowledge, and that, I think, is one of the most powerful things about service-learning with as well as engaged scholarship with.

John:That group that was doing the analysis of the data… Were they faculty? Were they students? Was it some mix?

Khuram: It was some mix, but here you do have kind of a hierarchy of knowledge and skill, I should say, in terms of how to do this, and so students and community members were trained by ethnographers and researchers on how to hold a tape recorder, what kinds of questions, and how to ask questions, the ethics of confidentiality, and then they went out and they conducted (after receiving a few weeks of training) these interviews in the community and it was the researchers, mostly faculty, that then booked and analyzed that data and ultimately synthesized that data, but every turn there was some part of this that was democratic and collaborative. Even the questions themselves were questions that the participants generated in concert with other community members. What is it that we want to know about ourselves? And so those were the questions that were ultimately used when we did the broader interviews.

Rebecca: Sounds like a really powerful way of breaking down the town-gown divide that happens in a lot of communities where there’s an institution of higher education.

Khuram: I think that it was transformational for all of us. I don’t think anyone could truly have appreciated what was going to happen, and I think part of it is because it was an open conversation and we sustained a certain level of openness, curiosity, and vulnerability to each other as well as what we hope would come out of it, and I mean for me it’s transformed the way I think about everything from teaching to service to even social action and the role of institutions of higher education in really engaging in communities, and so the power of it, I think, was also to reveal what’s possible that we are capable of operating on different terms and the institutions of higher education do not need to be paternalistic in their engagement with communities and they do not need to take a charity-based approach in their supportive communities; they can be collaborative, it just requires us to match strength to strength to define the things that are going to be valuable for college students and faculty and staff to learn from communities and what communities will benefit learning with their work with institutions of higher education.

John:It strikes me too that this type of project could be much more sustainable. Many service-learning projects or one-off projects where the students work and do something in the community or to the community or for the community, but when you get the community itself engaged it swould seem that that could, at least for some types of projects, set the stage for continued collaboration, either with later groups of students working with them or with the community itself. Has there been much success in continuing the efforts once the classes ended?

Khuram: I first off want to say that I absolutely agree that service-learning is conventionally structured as a one semester project-based or hour-based experience, and it’s usually focused on alleviating one particular social issue, and what we have found is that it’s necessary to do year-long initiatives and we’ve been very fortunate to see that this initiative has been able to sustain itself for over three years, but that’s required us to allow it to evolve into what it needed to and one of the biggest parts of that has been that it has been untied from any particular course. It used to just be tied to my classes and so students would do service learning project were tied to classes they were taking with me. Now, students are participating as participants in independent studies, they’re participating in different working groups that sustain themselves a little bit more autonomously, and that is also true for a lot of long-term community residents that have joined smaller working groups. There’s a working group on food insecurity, there’s a working group on political representation, there’s a working group on economic empowerment and economic opportunity, and so any one of these working groups becomes its own kind of autonomous community that intersects with long-term residents and college students and faculty and staff and that, I think, is a sign of progress and health, is when the institution of higher ed that’s tied to these projects doesn’t need to own it, control it, and manage every aspect of it. If it can become a little bit more fluid and have its own purpose outside of a predetermined purpose from the institution, it becomes more organic and more impactful often.

Rebecca: The continuity that set up in a structure like that of “community who doesn’t go away” versus students who drop in and out as they go through four years—they’re a member of the community but then they often leave—seems like it’s a really useful model for not only making the learning better but just making the impact better. Can you talk a little bit about the community’s response to these projects.

Khuram: Yes, drive-by service-learning isn’t the way to transform communities or students; it requires a real, authentic engagement, and I think when you put people in real situations you get real outcomes and that’s across the spectrum. So you’re going to get people that are going to collaborate, develop great friendships, but you’re also going to get friction and struggle and honest expressions of frustration with one another. And so that becomes a part of it too, so our students need to learn or end up learning—whether they need to or not—the ways in which their participation is both important but sometimes limited. They are going to sit and be witnesses to long-standing struggles in a community; for instance, long standing struggles between law enforcement and communities of color, and they’re going to find their own footing in those spaces; they’re going to need to make sense of how to be an ally, how to be an advocate for an inclusive community that they now belong to, so the stakes become a little bit more real. But I would be a little bit disingenuous if I was going to imply that it’s neat and tidy. I’ve received pushback at times. I remember we were holding a dialogue and I had said that we’re really starting to build some really empowering opportunities here and someone coughed and said, you’re from the colleges; you have all the power. It was a great check on my own assumptions about how I was being seen in that space… that participating in a community activity while still being associated in some ways representative of a very wealthy, multi-million dollar institution in a post-industrial Rust Belt City is not going to play out in someone else’s mind the way that it might in mine. Now what I’m proud of in that work is that someone felt that they were in a space where they could call out people’s unseen or unacknowledged privilege, and that I thought was really important for other people to see, and for me to experience, but it also means that tension in real relationships is ongoing. Honestly, we are not dealing with a utopian situation where we’re all playing on equal terms; we’re coming with different levels of capital and different levels of support within that community, so even as we do this work, my students are good to remember, as am I, we cannot be tourists in other people’s lives, that if we have certain privileges this is a place to take responsibility for some of them.

Rebecca: In a situation like this where tensions can be high, differences big sometimes, and you’re trying to dialogue, how do you set up that environment so people feel safe, like the situation that you’ve just described.

Khuram: Always sit in a circle. Always begin with some expectations. What do we need from each other to have respectful and productive and meaningful conversations? Let’s create those standards together and revisit them every time we sit in circle together. Have people that are prepared to facilitate, that have training or are getting training in facilitation; that needs to be, I think, a critical piece of that, because while it is important to hear from everyone, there is a lot of value in having someone who can reflect back some of the bigger messages and patterns that are emerging in the conversation, someone that can point to the standards that we’ve set for ourselves and what we expect as our best way of engaging, and to remind people that there are strategies that we’ve identified when things get really heated where we want to go with that. So, I think being very intentional about creating a dialogical space, and for us, the use of intergroup dialogue and a lot of the pedagogical strategies developed by the University of Michigan Intergroup Dialogue were very important and helpful resources to get started.

Rebecca: I think one of the things that I was hearing here that I want to just note, is if you’re having one of these conversations that you should have a facilitator and that the facilitator is not really participating in the conversation but rather facilitating the conversation. I think that can be challenging if we want to be involved in those conversations, but you need to make sure that you’ve picked that person and that person is staying as a third party.

Khuram: Yes, absolutely. And we typically have two people that will facilitate and that way there’s still some opportunity to give feedback or response or to slightly move out of a facilitator role, at least in terms of being able to share some ideas. But yeah, it does require you to pull back a bit. But having two facilitators… and it isn’t something that can’t be learned; I don’t think that people have to be lifelong professional facilitators. Most teachers are facilitators, and most of us have some experience facilitating or mediating conversations between others. As much as it’s important to start with people that have a background in facilitation, I think ultimately you want to end in a place where many of the participants feel comfortable and can contribute to the facilitation process over time, so we would meet every week. Ideally, we wanted to prepare people for their opportunity to do some facilitating. At this point we’ve seen dozens of participants go on to do much more formal facilitation in other spaces. That’s something that I’m very proud of and I’m very proud of them, I should say, for what they’ve accomplished.

John:You had mentioned some broad categories of tasks and working groups. What were some of the specific projects that were undertaken by people working in these projects in the community?

Khuram: All of these emerged dialogically as members of the campus community and long-term residents of the community talk through ways in which they felt connected and disconnected. We had four big ones, I’d say. We had community police relations, economic opportunity, food justice and food insecurity, and political representation. I’ll touch on each of them a little bit and then if you want to know a little bit more about any one of them I can pause. So, for food justice and insecurity, part of the challenge was an immediate one where it was about galvanizing community members to glean food and to increase access to fresh food, so we had volunteers doing gleaning. In the midst of that they were also looking at the president’s food deserts and dialoging along with community members about their access to nutrition and presenting some of those findings to the City Council and the Mayor. Or police community relations, we had two dedicated members who were part of a standing committee known as the Community Compact that met with different members of law enforcement and city government on a regular basis to talk about police-community relations and to develop programs to engage the community as well as to address certain policies. Then we have political representation, and for that what we saw was a wonderful volunteer energy of members of our entire group that went out and facilitated dialogues between political candidates and community members. Unlike conventional town halls where you’d have people sitting behind a table or behind a podium, we chat in circle with political candidates, and we had facilitators asking questions and facilitating dialogue in a pretty different kind of environment than I think a lot of us have when we engage with people that want to be elected, as well as elected officials. So we ran those, along with giving people an opportunity to register to vote. For economic empowerment, we trained facilitators to go out into the community in pairs and to hold circles in different corners of the community… in laundromats… in a variety of public spaces… to ask them what were the ways in which they were experiencing opportunity and what were the ways in which they were limited from economic opportunity. We also explored with them if they could wake up tomorrow to a different city, what would it look like? What opportunities would exist? And we took all of that and made it a final document called the “Big Talk in a Little City,” which has become an important and integral part of the city’s long-term commitment to economic empowerment, and so, not only are those voices and stories included in an official document, those voices and stories are now helping to shape policy and resource distribution in the city.

John:How have students reacted to this? Have any of them considered careers as working with communities and such things?

Khuram: For some of our graduates this has been life-changing. I think that one of the most fundamental things that we did well was simply to put people that would otherwise never have encountered each other in the same room and to ask them to share their stories and to talk about themselves. Developing those personal relationships between people that would otherwise pass each other on the street without a glance. People that had age differences, 40, 50, 60 years, people that had racial and socio-economic differences and geographic differences were suddenly having dinner at each other’s table, knew the names of family members, and knew the smallest things about one another were coming to their respective graduations and ceremonies and really becoming participants in each other’s lives. So, for a lot of our undergraduate students, having an opportunity like that is so deeply transformative because now policy is not just a matter of abstract equity and justice; it’s a matter of empathy and equity. You feel differently for someone who feels like a friend or family when they are in need and that informs your approach to policy and your approach to work in a community differently. So, we’ve had students that have gone on to do some really powerful work in law clinics, AmeriCorps and have stayed in the community to do some of that work because it was so transformational and they committed so much of their learning to this kind of engagement that they want to continue it. We do have a few folks that took a gap year between graduate school and stayed on, or decided to pursue a different kind of professional path because of the work they did.

John:That’s impressive.

Rebecca: It’s really exciting.

Khuram: I wouldn’t believe it if I hadn’t seen it.

John:Could you give us some idea of the scale of this—how many students are involved and how has it grown?

Khuram: We started with a relatively small group of about 20 students and 20 long-term community members, and in terms of active participants, it never really went much bigger than that, but it sustained itself over time and it also engaged a lot of other students and long-term community members for months at a time. What I mean by that, for instance, is a lot of our sustained participants would engage their friends, their roommates, their neighbors to come to our weekly sessions. So, we would oftentimes have topical session that were open to the public and those open sessions we could have up to 60, 70, 100, 200 people at those sessions, and so we had an active presence for quite a long time in the community when the courses were running, and now that we have the working groups there’s smaller numbers, but again, their impact, I think, in some ways is deeper because they’ve sustained some really deep work. One of the most incredible things that I saw the students do was they developed a course that would involve high school and college students learning together; so they essentially wanted to do what we were doing through these community dialogues in the high school. They wrote a course proposal, they submitted the course proposal, and after a few revisions and edits it was approved by both the college and the high school and we had a small group of about a half-dozen college students and a half-dozen high school students that took a course together at the high school. And that’s not a lot of people—but that doesn’t—what an incredible experience that they’re participating in something they helped codesign in order to address an issue that they perceive to be real across these age differences and community differences; that these teenagers and these college students together identified this town-gown divide and saw high school and college as a way to build bridges and constructed a course to do that and then participated in that course together. To me, that’s a kind of deep, transformative, impact that doesn’t quite reflect big numbers, but big experiences.

John:It’s certainly a testament to the impact that it had on those students that they were willing to do this and interested and motivated to do this.

Khuram: Absolutely.

John:How have your colleagues responded?

Khuram: I think that my colleagues have been excited, and I think that for many of them it created a new opportunity for them to engage. So, we’ve had faculty that have come in as participants, we’ve had them lead certain workshops and activities. They’ve come in with their expertise within their respective disciplines and fields. So, we’ve had a really great showing of faculty support. And part of it is we did not host this work on campus. We were very intentional about finding a place and space that was both a place that could be shared as well as a place that was easily accessible for long-term community residents, and so we found ourselves at the oldest black church in the city and a place that many of my colleagues had never been… that many people in the community had never been, and it was in the part of the city that is still segregated across a number of lines of race and class, and yet it was one of the most racially and socioeconomically diverse spaces that you could sit in in the city and here it was in a historically or at least currently segregated space… and so I think the opportunity for faculty and for staff to engage with a community that they’re really caring about in a context that seemed more inclusive was really exciting and affirmed a lot of their values. I think this is something that people really want, it’s just a matter of creating the opportunity so that they can engage in it. I don’t think that most faculty or staff want to engage in these kind of vertical relationships with communities. It’s just how we’ve been doing things for so long.

Rebecca: Seems like your background in teaching about equity and teaching about intersectionality and doing some research in the classroom about these topics set you up really well to do this work. Are there tips or other things that could provide faculty who don’t have that same background that you could share to give us a doorway in?

Khuram: I think that in some ways having a background as a scholar in any kind of social justice or equity field can be a barrier, and here’s why. That work is always in your head and it is disembodied in the institution, and the institution is, by its very nature, disembodied from the communities that it surrounds. And so you can very easily be a deft and prolific scholar of social inequity and convey and facilitate inequity in your actual life. So really it’s not a guarantee of anything. I think the measure of your capacity is in the doing, and I think it’s really about addressing questions. Who am I inviting to the table? Where is the table? Who is not here? What do I need to ask now to get who’s not here, here? Those are the more important questions, and I think if we don’t presume that there’s a certain kind of institutional privilege that comes even with being able to wax philosophical about questions of equity, then we’ve already lost the plot. We’ve got to honestly think about the spaces and places in which we’re doing our work and the kinds of privileges that we need to interrogate about ourselves before we can do any of this work in equitable and meaningful ways, and so I would say this work is for everybody, and this work is for anybody who is willing to really work with community members and to find shared purpose with community members. It’s willing to listen and learn from… and is not just interested in providing to.

Rebecca: Those are such great reminders… and empowering to make sure that we can all find a way to help and work with the communities that we live in.

Khuram: Yeah, and sometimes it does mean maybe rethinking a service-learning project that’s a semester long and seeing if you can map it out over a year. Would you spend a semester just creating relationships between students, yourself and long-term residents of a community just in that exploratory project? and then say, “Ok, out of this what have we identified collectively as a community need that we can address as a class?” …so that you get, of course, that buy-in, which is so important, but there’s a truly transformative possibility that is emerged that simply wasn’t there until you took the time to really connect and build that relationship, so I’m also in practical terms a really big proponent of year-long service-learning initiatives and moving away from the pressures of a semester-long initiative, unless you’re willing to do half a semester of really just relationship building and collective meaning-making and then cut the service piece a little shorter.

John:We usually wrap up the podcast with a question: “What are you going to do next?”

Khuram: What I would like to do next is to start preparing and supporting students to be the initiators of this work. I am currently working with a couple student groups that are creating their own curriculum and their own activities to engage people in the community with. Right now it’s a youth-to-youth, college student and high school student initiative, and the aim there is to just be a guide on the side, to really maximize whatever space and context I can help create for students to develop their own initiatives for engagement. Again, along these principles of working with, but to see our students become the guides that they need that our students can be the leaders that they’re looking for and that they can help develop leadership in their communities, and so for me right now what that involves is again having college students and high school students connect and collaborate and learn from each other with really very little use of faculty and take from us what you need and build what you must.

Rebecca: Sounds really exciting. Thank you so much for all that you shared today; I think it gives us all a lot to think about. Not just think about it; we need to take action too. [LAUGHTER]

Khuram:Thank you.

John:Thank you.
[MUSIC]

John:If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Theme music by Michael Gary Brewer.

46. Creative risk-taking

When you teach the same classes every year, it’s easy to fall into routines. Classes, though, can be much more fun for you and your students if you are willing to take some risk by experimenting with new teaching approaches. In this episode, Dr. Wendy Watson, a a senior lecturer of political science and pre-law advisor at the University of North Texas. joins us to discuss how she has engaged her students by introducing some very creative and fun assignments in her classes.

Show Notes

  • Ishiyama, J., & Watson, W. L. (2014). Using Computer-Based Writing Software to Facilitate Writing Assignments in Large Political Science Classes. Journal of Political Science Education, 10(1), 93-101.
  • Watson, W. L., Hamner, J., Oldmixon, E. A., & King, K. (2015). 14. After the apocalypse: a simulation for Introduction to Politics classes. Handbook on Teaching and Learning in Political Science and International Relations, 157.
  • Wendy Watson (2016) Best and Worst Teaching Moments (Center for Learning Enhancement, Assessment, and Redesign, UNT video) – This contains a description of the zombie apocalypse project.
  • Center for Learning Enhancement, Assessment, and Redesign at UNT
  • Olson, Katie (2017). “Local Author Gets Cozy with Mystery Genre.” The Dentonite. October 3, 2017
  • Wendy Lyn Watson – author website

Transcript

John: When you teach the same classes every year, it’s very easy to fall into routines. Classes, though, can be much more fun for you and your students if you are willing to take some risk by experimenting with new teaching approaches. In this episode, we examine how one professor has engaged her students by introducing some very creative and fun assignments in her classes.

[Music]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.
Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[Music]
Rebecca: Today our guest is Dr. Wendy Watson, a senior lecturer in political science and pre-law advisor at the University of North Texas. Welcome, Wendy.
Wendy: Hi, thank you for having me.

John: We’re glad to have you here.
Our teas today are:
Wendy: I am drinking Paris. It’s a blend from Harney and Sons.

John: We have that next door.
Rebecca: Yeah, a tasty one. I have Irish breakfast tea today.

John: …and I have ginger peach green tea.
We invited you here to talk a little bit about some of the interesting things you’re doing with your classes. Could you tell us first a little bit about the classes that you normally teach.
Wendy: Sure. In the state of Texas there is a requirement that every student take two Introduction to American Politics courses in our department. We refer to that as the full employment plan. So, I teach both of those courses and then, other than that, I teach all of our law related courses. I’m not the only one, but I teach all of the law related courses: our legal systems course, civil rights and civil liberties, the rights of criminal defendants, constitutional law, an LSAT prep class, gay rights in the Constitution, and a seminar on the death penalty, in varying cycles.

John: You do quite a few innovative things in your classes, and one of those is having your students rewrite the Constitution after a zombie apocalypse. Could you tell us a little bit about that activity?
Wendy: Yeah, the idea is that the zombie apocalypse has occurred. This is actually for one of the flavors of Introduction to American politics, and this particular course deals with institutions: the founding of the Constitution, federalism, the legislature, the executive, the judiciary, and civil rights and civil liberties. The idea is that the zombie apocalypse has occurred. Huge portions of the population of the US have been destroyed and the remaining members of the country are required to rebuild the United States and part of that is rewriting the Constitution. Essentially, what they’re doing is building a government from the state of nature, but they don’t know that. They think they’re building a constitution after the zombie apocalypse, and that’s way more fun. It’s a guided exercise; they get worksheets every week making them think about “What is bicameralism? What are the benefits of bicameralism? What are the drawbacks of bicameralism? etc. They don’t just get to go off and write crazy things. They actually have to think about stuff and then they work in groups to create these Constitutions. One of the things that I really love about the course is that they actually do have to grapple with these issues. They sometimes get pretty heated.

John: How large are the groups?
Wendy: Usually these introductory courses are about a hundred and twenty five students and I put them in groups of about five to seven.
Rebecca: And are these things that happened outside of class, in class, online?
Wendy: No. I’ve taught the class as an online class in which case it obviously happens online, but when I’m teaching the class as a face-to-face course I actually do give them class time. Having them do it outside of the class nothing ever happened, giving them the time in class keeps them from hating me and also ensures that they actually do provide some sort of useful product at the end.
Rebecca: What assignments or exercises or things that you would normally do in class does this exercise replace?
Wendy: You know it doesn’t actually replace any exercises because if I weren’t using this activity, all of their homework would be outside of class and they’re still doing all of that. So what it’s really replacing is me lecturing and I’ve got no problem with that and I don’t think they have a problem with that. It’s more exciting or more interesting for them to be doing something, talking to each other than it is to be sitting in a seat listening to me, I think, I’m pretty sure. And I think it’s actually more educational for them to be engaging in the material as opposed to passively sitting and listening to me. Yeah. So although all they’re missing out on is me talking.
Rebecca: How did you how did you decide to go in this direction and develop this particular activity?
Wendy: I was trying to think of a way to create a simulation that would last throughout that semester, so something that kind of continued over the course of a term. And I wasn’t really sure what that would be, and I think we were watching The Walking Dead. But honestly how that all came together I couldn’t tell you, but yeah I’m really happy with that. It’s been adopted by several of my colleagues and by a professor at University of Whitewater. She used it in a summer program for high school students, and yeah, I’m really happy with that how that one turned out.
Rebecca: How did the students respond?
Wendy: You know,of course there are always students who are not going to respond at all. But I’ve never had a student who actively said that they hated it which is, I guess, good. I’ve had a lot of positive feedback and I had you know one of my best student interactions ever over this particular assignment. Again, I’m going to apologize to all of my biology friends out there. One of the features of the assignment is that the zombies fall into two categories. The type one zombies who are traditional brain-eating zombies and then the type two zombies who have developed a lesser mutated form of the virus. And so they have features of zombies, they have the shuffling gait and the slurred speech, but they have higher order cognitive thinking and they don’t eat brains, and they’re just generally safe. But if two type two zombies have a child together there is a probability or a possibility that their child will be a type 1 zombie. Again, this makes no sense at all, since it’s a virus and that just doesn’t make any sense. But, it raises this question of what do you do with type 2 zombies? Do you sterilize them? Do you kill them? What do you do with them? And they were grappling with this issue one day. And this poor student comes in, and he was, I swear to God, he was almost in tears. Because his group had decided collectively to exterminate the type 2 zombies and he said,” what do you do when you encounter people who are terrible?” And so he ended up having this long talk about how do you deal with the notion that there are Hitler’s in the world. I was like “Well, you have to remember that there are Gandhi’s in the world.” It was a long and lovely conversation about the essence of mankind and the balance of good and evil. And I kept emphasizing to him that this wasn’t real and that his friends were not evil, but anyway it was it was a great conversation and I was so touched that he took it so seriously. It’s just a testament to me of the fact that students really are interested in the material if you give them an opportunity to be interested in the material.
Rebecca: It sounds to me too like it allows them to really grapple with the really difficult conversations that are around rights and lack of rights and who gets those rights. That might be really uncomfortable if you talk about it in a in a real situation, but in this safe simulation you can have some of those challenging conversations that you might not be able to have as effectively.
Wendy: Yeah, I think that’s right. If you’re talking about things like race or sexual orientation, you’re always confronted with the fact that there are people in the room whose actual rights are implicated, and that does tend to make people sent to themselves perhaps, and that’s not necessarily what you want in real active discourse. So, when you’re talking about something that is seemingly unreal, it is unreal… they’re zombies… it’s not real. I do think that it gives people the opportunity to think through issues in a way that is safer, but also more honest.

John: The type 2 zombies add to the degree of difficulty or the level of challenge there.
Wendy: Yes, exactly.

John: You’ve also created a 500-person learning community, could you tell us a little bit about that?
Wendy: Yeah, that was nuts! My university decided to try to create a variety of different models of learning communities, sort of all at once, that alone was nuts. But I was going to be involved in a combined course learning community, so without any residential component. And I found this wonderful man in the psychology department who probably had no idea what he was getting into, and we created this community that was 500 students. His Introduction to Psychology course and my Introduction of Political Behavior course, that’s the other half of our introductory American politics duo. And our courses were back-to-back, so there were times when he could have two hours, and times when I could have two hours. And we focused on political psychology, specifically as it related to campaigns. And over the course of the semester, they each had to read three or four articles and write one page papers about them, little summaries, and then they came together and they shared their information, and they had to come up with the campaign strategy for either one or two presumed political presidential candidates. At the time we thought that was going to be Clinton and Rubio… that obviously didn’t happen. But they created these poster presentations and then we picked from among those poster presentations the 10 best, and we took those to UNT on the square which is a little gallery space in downtown Denton. And we invited faculty and university administration and we invited the Denton Record Chronicle which is our local newspaper. And the students really got into it, the ones who won showed up with their little red bow ties if they were representing Rubio and they had candy at their stations. And it was really awesome. It was great.
Rebecca: What do you think one of the biggest learning gains was for students who were in this learning community scenario where you were diving into something in depth from two different points of view?
Wendy: I think one of the things that they gained was an understanding that these two disciplines actually interacted with one another, that psychology and political science weren’t sort of siloed ideas, that they actually were related to one another. And I think one of the other things that they learned is that what they learned in class actually had implications for the real world. That things that we were learning in psychology and political science had implications for how politicians were actually running their campaigns. And that they could take the skills that they were learning at UNT and potentially apply them to a job, which is always a big thing. [LAUGHTER] Getting a job is good.
Rebecca: What level are the students in these classes?
Wendy: In those particular learning communities, most of the students were freshmen, first-year students, because they had to be advised into them, somebody had to sort of point them towards this pair of courses, so they tended to be freshmen. Otherwise these courses actually tend to draw students all the way up to their senior year, because they put them off until they have to graduate. But for these particular communities, they pretty much have an advisor say, “Hey, here’s a good idea. Take both of these courses.” They tended to be freshmen.
Rebecca: Did you find that the learning community method works particularly well with first-year students?
Wendy: I think for a lot of types of innovation it doesn’t necessarily, but I think for this, it did, because I think their desire to please was strong. And I think that they didn’t any preconceived notions of what college classes were supposed to be like, so they were maybe more receptive to the idea of doing something different. For all they knew this is what it was supposed to be like. [LAUGHTER]

John: …and getting that introduction to an interdisciplinary view of the world is probably good to do before they get too deeply into the silos of their major.
Wendy: Yeah, I agree.
Rebecca: So you’re full of brainy ideas and another one that you pulled off was an online Electoral College simulation game, can you tell us about that too?
Wendy: Yeah. So that was a lot of that was a lot of fun. I actually have to give most of the credit for the online component to our office, here it’s called CLEAR the Center for Learning Enhancement Assessment and Redesign. The assessment component has largely gone out of clear, but that’s still what we call them. They do all of our online support, learning management system, redesigned helping us create online courses, all of that sort of work. And I had a sort of a low-tech version of this course. Originally they were working in groups, I always make them work in groups, I don’t know why. But they had groups and the idea was I used the map from 270 to win, which has sort of the baseline Electoral College predictions, and which states are going red, which states are going blue, etc. And then students had campaign money and they could essentially bet their money on individual states. And if you were the Republican Party and you bet fifty dollars here, but then the Democrats get 51, then the Democrats won the state, so whoever bet more money in a state won the state. And so you could see the strategy of betting in different states of spending more campaign money more campaign resources in each state, and as you won a state, the states that were blue moved around or the states that were red moved around and you could see the total – who was winning the electoral college. And it was played in three rounds. But this was a huge pain to implement in the classroom with having to update this Excel spreadsheet every round and get people’s votes every round. It was a nightmare. So CLEAR created an online version for us that allowed students to play against each other online and it was really slick, it was beautiful, I loved it.
Rebecca: So, I’m noticing the “loved” as opposed to “I love it”.
Wendy: You notice that didn’t you.[LAUGHTER]…… Yeah, so I think another point to make here is that if you’re going to launch into one of these grand plans, you really do want to have some long-term commitment from your University. I love my university but long-term commitment is not their forte and for the learning community, for example, Adriel and I (my co-conspirator and I), we put a lot of effort into that course and we ended up offering it twice. It went really well both times but to the extent we needed money it came from a Title III grant that ended. So, we didn’t have the money anymore and then we also depended very much on help from the registrar, from advising, and from admissions to help us coordinate all of the the details. Because it was no small matter, right? It was actually very difficult. It wasn’t just us. There are all sorts of offices that had to help us out with this. And the university basically was like, “Oh, we’re done.” That was difficult and so we just lost the necessary institutional support for maintaining that program. And with the electoral college I went for like a year and a half without teaching that course, so it didn’t get used because nobody else was using it. And so CLEAR stopped supporting it on their website. It just went away and it’s just gone. So, it’s just one of those things. You kind of need to get it in writing, because there’s a tremendous amount of start-up costs associated with these programs and unless you know that that’s going to carry forth and this investment is going to pay out over an extended period of time, tt could be a little bit demoralizing.

John: In one of your other experiments in class, you did something with a mystery room. Could you tell us how that worked?
Wendy: Oh yeah, that was this last year. That was so much fun. Yeah, so the game was actually called Free Lucky. Lucky is UNT’s unofficial mascot. He’s an albino squirrel; he’s actually not lucky at all. We’ve had a series of Lucky’s on campus and the only two that I’m aware of… one got carried off by a red tail hawk and the other one got hit by a car, so they’re not lucky. [LAUGHTER] But we call him Lucky and you can get little lucky dolls. And so I got little Lucky dolls and I shoved them in little cloth pencil cases and I put combination locks on the pencil cases, so he had to get him out by undoing the lock. And I’m put my groups of students… groups again… in various study rooms in the library and they each had a little encased enshrouded Lucky in their room. And then they started the game with a question on their learning management system on Blackboard. This was for an LSAT prep course and the beautiful thing about the LSAT is that you have these questions with very specific answers. No question… here’s the answer… that’s it. The first question, if they got it right, it led them to a webpage with another question; if they got it wrong it led them to a webpage that had nothing and then it sent them back to the original page, and so forth and so on. It sent them to various pages around the web, some of them with clues, some of them with other questions, eventually it would’ve taken them off of the web and sometimes it pointed them to different clues around the room. There were various and sundry things on the table, some of them which mattered… there was a playing card… it actually was a clue, but then there were things like spools of thread that meant nothing. There was envelopes taped under the table that had a whole series of questions. And the questions there, if you answered them all, there were four of them and those gave you letters and then there was a tongue depressor on the table that helped you translate the letters into numbers and that was the code to the combination lock and that allowed you to free Lucky. And the first team that got Lucky to me… I was sitting in the lobby of the library, first team that got Lucky to me won… and they won packets of colored highlighters, which doesn’t sound exciting but they were all pre-law students and that’s like gold in the legal community… is colored highlighters. So it was exciting, they were really thrilled.

John: It sounds like fun.
Rebecca: It sounds like a lot of fun.
Wendy: It was.
Rebecca: What made you decide to do a mystery room?
Wendy: Well, you know, we have one here in Denton, and I think it looks really cool and I want to go, but I can never get people to go with me, and so I decided well I’m just gonna create my own. I wanted to do something, again, that was interesting. As much as the LSAT prep stuff was really interesting and important for my students, it’s not super engaging. We could stand up there and write logic game trees on the board, for hours on end, but that’s not exciting. That’s not even lecture exciting, that’s just really really boring. So I wanted to at least break up the class a little bit by having something that was more engaging, more active, something that was interesting.

John: And it brings in gamification too, where there’s some incentives and competition.
Wendy: Yeah. Oh yeah, the competition was big. I had one group that came down with Lucky after about a minute and a half. I was like, “You did not answer all those questions.” The guy who handed me Lucky, he’s like, “You gave this puzzle to a marine .” [LAUGHTER] I was like, “So, did you just bust the lock?” He’s like “No, I didn’t have to bust the lock. I could get him out without busting the lock.” I was like “You have to open the lock, you can’t cheat.” [LAUGHTER]. So they went back, they did it. But anyway, yeah, it was definitely a game to them. They were serious about it.
Rebecca: That’s hysterical and unexpected, right? [LAUGHTER]
Wendy: Completely.

John: A common theme of all this is that you seem to experiment with your classes and take some risks in trying new things. Could you tell us a little bit about what prompts you to do that?
Wendy: A couple of things. One, is that honestly it keeps me interested in the courses. I can get bored with the material as much as they can. In fact, they sit through it for a single semester, I sit through it for semester, after semester, after semester. And you can only talk about the appointments clause for one or two times before you’re like “Oh my god, I’m gonna dig my eyes out. This is really dull.” And that’s something I actually enjoy, right? I think the appointments clause is interesting. You still want to shake it up a little bit. And the other reason is that I really do believe that students learn better if they are engaged. As much as I love to hear myself speak, I don’t necessarily think that they love to hear me speak. I think that they get more out of my class if they are doing something. If they are seeing some connection between what we’re doing in the real world. If they can see themselves actively engaged. If they have a sense that they have power in the class. Some sense of control over their own education. I think all of those things are really valuable to them. So it’s a little more effort for me, but I think the payoffs are worth it.
Rebecca: So all of these examples that you’ve shared with us today are really different from one another: they use different technologies, different setups. What is your advice to someone who wants to take some risks and try something new, but it’s something that they’ve never done before?
Wendy: Start small. Don’t start with a 500 person learning community, which is what I did. That was dumb. It worked out, but it was dumb. Yeah, start small. Collaborate with somebody so you have somebody to lean on and share ideas. That’s maybe why the learning community worked, is that I had something called the Core Academy, so we were focused on these sorts of things together. And then I had my my co-teacher, Adriel, to work with. I think having a support system and starting small is the way to go. You don’t have to do a semester long simulation, you can devote one class to something. Use a method that lots of people are using, like team-based learning. You don’t have to do that all semester you could do it for one class. There’s nothing wrong with starting small and then getting bigger.
Rebecca: Did you start small?
Wendy: I did not [Laughter].

John: Somehow I suspected that would be the answer.
Wendy: Yeah, that’s not my style. But again, I think that if you’re worried about getting started, if you are less stupid than I am, then don’t hesitate to start small. There’s nothing wrong with that.
Rebecca: Have you had any student resistance to some of the alternative or non-traditional methods that you’ve been using in your classes?
Wendy: I get a little resistance sometimes. For the most part, they actually seem to enjoy it. Every now and then I’ll get a student who seems to think that I’m not doing my job. I mean I’ve had students who flat out on evaluation have said “I expected to come to class and hear you talk and you didn’t.” Like “Really? That was what you expected?” I mean, yeah, I assumed that is the expectation, but like, “You’re disappointed that didn’t happen?” I can’t imagine that. And of course there’s always, as I mentioned, a lot of these things involve group work, and a lot of students have resistance to group work. Even when the group work ultimately works out okay, they still are annoyed that I put them in groups. Just the anxiety associated with group work carries over to the end of the semester. Of course, some groups don’t work out. You’ve always got somebody in some group that either doesn’t pull their weight, or is responsible for a part of the project and fails to turn it in, or somebody in the group who is bossy. You always have some group that’s got a problem and I usually try to mediate that situation, but sometimes they don’t come to me until it’s too late. There are always points of contention. But they’re relatively few, and honestly I’ve always got a few complaints when I lecture too. I’d like to say I never have complaints there, but I do.
Rebecca: I read this really great article about you being a mystery novelist.
Wendy: I am.
Rebecca: Can you talk a little bit about that?
Wendy: Sure, yeah. I am a mystery writer. I started writing a long time ago, right around 2001 actually. A bad year. But I had my first novel published in 2009. I write a type of mystery called a Cozy, which is exactly what it sounds like, it’s cozy. They are light, often funny mysteries. Amateur sleuths, so no cops or private investigators. They can be in the book, but they’re not the primary character. Usually female sleuths, small town, no sex or violence on the page. I mean obviously somebody dies but it happens off the screen. They’re really quite delightful. I said PG-13, I actually included the word “bitch” in my first book, and it wasn’t even calling somebody bitch. It was like son of a bitch. I hope I’m not destroying your podcast by using that word [Laughter]. I actually got nasty emails about using that word. Really? Oh my goodness. I don’t use that word there anymore. Yeah so I started writing I’m working on number seven right now and, that’s that.

John: How do you manage that along with all your innovation in class? It seems like that’s a lot of demands on your time. How do you allocate your time?
Wendy: Not well. Yeah, I was talking about this with a colleague this morning, we were talking about this LSAT prep course (she’s teaching it this semester) about the fact that prelaw students really should be studying a lot for the LSAT. It’s a huge portion of their application. Yet, for some reason, they don’t and instead they focus so much on their GPA, which is important, but honestly, not as important as their LSAT score. They shouldn’t let their GPA slide either, let’s be clear. But in the grand scheme they should be focusing on their LSAT score. We were discussing the fact that the LSAT is way far away but their GPA is right in front of them, and so that just feels like the thing they need to tackle right now. And for me my deadlines are way far away and my courses right in front of me. So I tend to focus on my coursework and I’m not so great about meeting my deadlines, and I apologize deeply to my editor, but that’s just the way it is. I do though have a calendar, a very detailed calendar, that I keep, that has specific time set aside for every single thing that I do. Not always true to that calendar, but I do have a calendar, and it includes time set aside for writing.
Rebecca: Do you find that your writing life and your teaching life influence one another?
Wendy: Yes. Certainly my academic life has influenced my writing life. One of my books was set on college campus and I got to kill off a couple people that I didn’t like so much, which was awesome [Laughter]. Certainly, I think that my tendency toward narrative, toward storytelling, influences my use of hypotheticals in my classes. To the extent that I’m sort of telling stories. Like the zombie apocalypse, I didn’t just write a paragraph: there has been a zombie apocalypse. It’s this, probably too long story, about this has happened, and it’s all dramatic, and that’s definitely a carryover from my writing life.
Rebecca: I imagine that those details though and that spike in the climax to a story, are all the things that get students really engaged and interested and and buy into the simulation and take it seriously. As opposed to something that’s a little more surface level and that it’s a little harder to imagine.
Wendy: Yeah, and I think sometimes one of the things my cozies tend to include is humor, at least I hope it’s humor. I tend to inject that into my hypotheticals a lot and I think that that helps. One of the simulations that I do in my legal systems class is a negotiation divorce case. Each side in the negotiation has information about their client. Some of its common knowledge, that both sides have, and the wife’s attorney has knowledge that only the wife has provided and the husband’s attorney has information that only the husband has provided, and they know that that information is going to come up during the negotiation in a series of PowerPoint slides. They don’t know when that’s going to happen, but the idea is that all the sudden the wife is going to blurt something out during the negotiation. They also don’t know that there’s information that the husband and wife have not told their own attorney and that’s going to come out in the course of the negotiation. So I had great fun crafting the simulation; like the things that the husband and wife have done, and the pieces of information that come out are delicious, and the students have so much fun finding out about these details. And yeah, I think that that makes the whole simulation so much more engaging, instead of just calculating the appropriate alimony. I think it’s a lot more fun.
Rebecca: Can you share a couple of tips from your creative writing self that might help other people come up with hypotheticals or examples that they could use in their classes?
Wendy: Yeah, I think one thing that you want to do is provide detail. If you’re going to create a hypothetical, create a character to go with the hypothetical, and then provide some detail about the character and the setting and those sorts of things. It really enriches the hypothetical. It doesn’t all have to be completely relevant. In fact, sometimes it’s better if it’s not all relevant because then it forces the student to look past the things that aren’t relevant to find the things that are. I think that’s probably the key is to include at least one person in your narrative and then provide some detail. Provide a setting, provide some description of your character, provide some element of detail about what’s happening, so that it’s not sterile or clinical. Because that’s, like you said, that’s really going to draw the student in, in a way that’s sort of, A happened, B happened, C happened, or not.
Rebecca: That’s great advice [LAUGHTER].

John: We always end with the question, what are you going to do next?
Wendy: So this year I’m actually not teaching, which it is really weird for me. Last year this time, I took a position as the director of the university’s core curriculum. So, this year I’m going to be continuing with my pre-law advising but otherwise I’m focused on the university’s core curriculum. I will be engaged in assessment, which is everybody’s favorite thing, but I’m also gonna be developing a lot of programs related to our cores. So some programs related to writing across the curriculum, some programs related to bringing back, I hope, some of our learning community endeavors, and possibly exploring some other options that would allow us to really enrich our university core curriculum for our students. When I talk to students now they talk about them as the basics or the things that they have to check off, and I want them to think of those classes as something more than that. So that’s what’s next for me.
Rebecca: Sounds like the right person might be in that job to help inspire students. [LAUGHTER] I think sometimes that’s a hard sell these days, helping students recognize the value of a liberal education, and get them excited about it and help them find connections.
Wendy: Yeah, I agree. I think I have a tough road ahead of me but I’m going to do my best.
Rebecca: I look forward to hearing more about it.
Wendy: Yeah, thank you. I’d love to come back sometime.

John: We’d love to have you back.
Rebecca: Thank you so much for spending time with us this afternoon and sharing all your great initiatives in your classes, I hope it’ll inspire a lot of our listeners.
Wendy: Thank you, I really enjoyed it.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Theme music by Michael Gary Brewer.

45. Opening the STEM Pipeline

Preschool through high school experiences have a direct impact on the majors and disciplines that students want to study and engage with in college. Designing these experiences to invite underrepresented groups into the discipline early can help to inspire and motivate a new generation of professionals. In this episode, Dr. Stacy Klein-Gardner joins us to discuss how engineers are attempting to diversify the field.

Show Notes

Related publications:

  • Parry, EA, PS Lottero-Perdue, SS Klein-Gardner.  Engineering Professional Societies and Pre-university Engineering Education.  In M. deVries, L. Gumaelius, and I.-B Skogh (Eds.) Pre-university Engineering Education.  Rotterdam, Netherlands: Sense Publishers. 2016.
  • Reimers, J. E., Farmer, C. L., & Klein-Gardner, S. S. (2015). An introduction to the standards for preparation and professional development for teachers of engineering. Journal of Pre-College Engineering Education Research (J-PEER), 5(1), 5.
  • Klein-Gardner, S. S., Johnston, M. E., & Benson, L. (2012). Impact of RET teacher-developed curriculum units on classroom experiences for teachers and students. Journal of Pre-College Engineering Education Research (J-PEER), 2(2), 4.
  • Klein-Gardner, SS, ME Johnston, L Benson. Impact of the RET Teacher-Developed Curriculum on their teaching strategies and student motivation.  Journal of Pre-College Engineering Education Research. 2(2):21-35. 2012.
  • Faber, C., Hardin, E., Klein-Gardner, S., & Benson, L. (2014). Development of teachers as scientists in research experiences for teachers programs. Journal of Science Teacher Education, 25(7), 785-806.
  • Mckay, M., Klein-Gardner, S. S., Zook, K. A., Yoder, M., Moskal, B. M., Hacker, M., … & Houchens, B. C. (2011). Best Practices in K-12 and University Partnerships Panel Winners ASEE K-12 and Pre-College Engineering Division. In American Society for Engineering Education. American Society for Engineering Education.

Transcript

Rebecca: Preschool through high school experiences have a direct impact on the majors and disciplines that students want to study and engage with in college. Designing these experiences to invite underrepresented groups into the discipline early can help to inspire and motivate a new generation of professionals. In this episode, we explore how engineers are attempting to diversify the field.

[Music]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[Music]

John: Our guest today is Dr. Stacy Klein-Gardner, the founding director of the Center for STEM Education for Girls, and currently an Adoint Associate Professor of Biomedical Engineering at Vanderbilt University, and a Senior Professional Development Provider with Engineering is Elementary, at the Museum of Science in Boston. She recently was appointed as a Fellow of the American Society for Engineering Education.
Welcome, Stacy.

Stacy: Thank you. I’m delighted to be here.

Rebecca: Welcome. Today our teas are…

Stacy: Well, I have to confess that I don’t care for tea. So, I had some lemonade with lunch and I’m good to go now. [LAUGHTER]

John: I’m drinking Tea Forte black currant tea.

Rebecca: …and I’m having Lady Grey today.

John: We’ve invited you to join us because of your very extensive work in improving educational, P to 12 STEM and STEAM education pathways in many ways. First, though, could you talk a little bit about your own pathway to a career in engineering and engineering education?

Stacy: Sure. I’d be happy to. I grew up in the American South…actually went to junior high and high school in Oxford, Mississippi. I wasn’t always satisfied with my educational opportunities there, so I spent every summer possible at the Duke University talent identification program, or Duke TIP. Which is where I made some wonderful lifelong friends that have influenced my personal life and career since then. I did go to Duke University, where I double majored in biomedical and electrical engineering. I spent my summers working at Duke TIP, really falling in love with education and realizing my passion for that. I did a masters and a PhD in biomedical engineering from Drexel and Vanderbilt University, respectively. Then, I always thought I would retire to teach high school one day and realized that was stupid, and if that’s what I really wanted to do, I should go do it. So, in the same Fall, I defended my dissertation, I started teaching high school full time and fell in love with being in the classroom and working with teachers. Since then, I’ve been a high school teacher, full or part-time, for over 20 years now and I’ve been on the Vanderbilt University faculty since 1999…and I’ve done everything from being Associate Dean to research track professor to adjoint professor now…but really have enjoyed creating my own career in engineering education.

Rebecca: You mentioned being the Associate Dean for Outreach at Vanderbilt School of Engineering. Can you describe what your role was like? I think it’s a little unusual, perhaps, to have an outreach dean so I think it’d be interesting to hear about that.

Stacy: Yeah, the title was definitely unusual at the time. You do find more positions now, often maybe at the assistant level. But, I had a really diverse group of things I was in charge of. I worked with our Career Center on setting up appropriate opportunities for the undergraduate and graduate engineers coming out. I managed a big sponsored lecture we had every year. My favorite part was definitely doing K-12 outreach for the School of Engineering and reaching out to local communities and schools and students. Another favorite part, one that maybe surprised me a bit that I ended up really loving, was study abroad for engineers and finding ways to help engineers figure out a way to get abroad. ‘Cause the rumor used to be that engineers couldn’t study abroad, but there’s so many more types of programs that you can go to and so mine was, finding the right kind of programs and aligning those with the degree requirements of Engineers and then helping the engineers know how to plan ahead to actually travel on them.

Rebecca: So, can you talk a little bit more about your work in K-12 and also the study abroad stuff because in fields where we might not usually think about these as being good matches, like engineering, we’re always looking for new strategies to find those relationships and what have you. So, can you talk a little bit about some of the strategies or things that you developed?

Stacy: Sure. In study abroad, a lot of it was doing the logistics, but some of it was also helping engineers realize that in order to come up with good engineering solutions, you have to really understand the client for whom you’re working; the person who’s found the problem that you’re looking to solve. So you need to not just understand the straight up science, technology, engineering, and math, but you also need to understand the culture of the person, perhaps the language…What is it about their environment that makes different design constraints? So, I think, having engineers study abroad, in such an international world that we live in, is crucial now. I’ve really seen it grow in popularity which has been really fun, even though I’m not in charge of it anymore. We have a very high percentage of students at Vanderbilt who now study abroad as engineers. The second half of that question, or maybe I took him in out of order, was around K-12. You know, at the time I was doing a lot of funded work by the National Science Foundation. My favorite project was a Research Experiences for Teachers program (RET). This is a program where you bring, typically, high school faculty (although that’s broadened some since then) onto your university campus, for six weeks during the summer. Then I would place those teachers into different labs that I had picked very carefully and they would have an assigned project that they worked on full-time, for most of those six weeks…and then at the end of that time, I would work with them on designing curriculum that would be both standards-based (so they would be allowed to teach it in their classroom) as well as based in the research of their labs. So that they were bringing in real-world, current research that was going on, and often the people from that lab would come to the high school as well. Then we would publish those units through a wonderful national digital library called TeachEngineering.org. So that was definitely my favorite piece. I did some other work. I designed some high school level medical imaging curriculum units, and getting to where people have a better grasp of “what is ultrasound? or MRI? and how do those things work?” and actually motivate you to want to study high school physics or math or something like that.

Rebecca: That sounds really exciting and a great way to get people involved in fields they might not know that much about.

Stacy: It’s definitely important, especially when you’re thinking about subjects that sometimes get a bad rap for being particularly challenging. It’s good to let people see why it is they’re learning those and to put that, when am I ever going to use that, upfront so they know exactly when and how they’re going to use that.

John: Has there been any follow up in terms of following students to see how many of them did go into careers in STEM fields?

Stacy: It’s a little hard to get some of that data because I often work at the teacher level and it’s a whole other level of IRB [LAUGHTER] to get at student level data.

John: That’s true.

Stacy: You know, I think it’s somewhat depressing in that the numbers for engineering percentage-wise aren’t increasing rapidly at all, even though a lot of people are putting a lot of time and effort into it. So, not always, I mean I definitely have a lot more confident teachers in the Middle Tennessee area who are integrating what is going on in engineering into their classrooms. Of course it helps now that the next generation science standards have engineering embedded into them and just recently in my state the Tennessee state science standards do as well.

John: In 2010 to 11, you established the Metropolitan Nashville Public Schools K to 12 Engineering Pathways. What problems did this address? How has it worked and are you still working with the Nashville public schools?

Stacy: I think one of the biggest problems it was created to address is a misalignment between what different careers and companies are looking for in their high school graduates, as well as probably colleges too, with what the high schools were producing. There’s such an emphasis now on STEM, and problem solving, and computational thinking that really wasn’t being addressed by the schools and so, with Race to the Top money, Metro Nashville Public Schools set out to form this engineering pathway. I was heavily involved with it for that particular year. I did a lot of professional development. I did all of the professional development for the elementary school that was part of this K through 12 pathway, using the Engineering As Elementary curricula and integrating that. Then, at the high school level, I actually co-taught a ninth grade engineering course at this particular school. So, I was helping another teacher who was an engineer by training but didn’t have as much of pedagogy, and that sort of thing. Trying to help her build up her skills and left her rolling. That high school, Stratford High School, is still clicking along and doing really well with STEM education. It’s growing in reputation and now has a middle school that’s been integrated into the campus as well. So, I think it’s it’s been a success and will continue to be. One of the teachers with whom I worked with the most there is now the STEM director for the entire district. It’s been nice to watch her come from being one of my RET teachers to that position at Stratford, now to leading our entire district. My involvement with the district kind of waxes and wanes over the years. You know, I’ll get really involved for a while, and then I’ll be less involved for a while. I’m not working with Metro Nashville Public Schools right now although I’m always available if they call on me for anything in particular. I’ve actually just joined an advisory board for the Williamson County Public Schools which is just south of here. So, I like to keep my finger in the pie in something locally, but then I often try to work more on a national level.

Rebecca: I wanted to follow up a little bit on what you were talking about elementary education and engineering. For many of us, perhaps, when we went to elementary school, engineering wasn’t a part of that curriculum. So, for those of us that aren’t in engineering can you talk a little bit about what that even looks like?

Stacy: I’d be delighted to. If you think about what the characteristics of an engineer really are…it’s around someone who’s creative, and who thinks outside of the box, and brings in different kinds of solutions, and doesn’t have a lot of preconceived notions. If anything, that’s exactly what a preschool to elementary age child does. They haven’t sort of been beaten down by the system to think in a particular way. They still have that inherent creativity. So, the ideal time to introduce the field of engineering is at the preschool through elementary levels…so that they learn what the field is about, can identify what an engineer does, and have positive feelings towards it, and that we’re creating them to be more STEM literate citizens. There are multiple programs out there. The one with which I’m most familiar, and have even liked so much I’ve joined their staff, is with Engineering is Elementary. But, with any of them you find an authentic but sort of compacted version of the engineering design process. I might look at what a college student would use, or even a high school student might use. and we might call out 12 different phases of the engineering design process. But, in elementary school we have five fingers, so we have five steps to the engineering design process, [LAUGHTER], and in preschool we have three steps. So, just kind of compacting it a little bit…always providing an accurate view of the field. Then giving the kids age-appropriate challenges, things that might happen to an elementary aged child, and then asking them to problem-solve.

Rebecca: Can you give an example?

Stacy: Oh sure. There’s one of the EIE units that comes to mind, where the kids are out there playing a sport and their team needs to be cheered on. They find this little turtle nearby, and they win the game, and so they decide that they’re gonna keep the turtle, and they have to bring the turtle back for the next round of the playoffs. Somebody’s got to keep the turtle in a place where the turtle can not die, because that would not be good for school spirit at all. So, the whole question becomes around, what do you need to design in order to have a habitat that this turtle can live in? They draw upon the appropriate science in this particular unit…and a lot of its around membranes and creating a habitat that has enough water but not too much water. So they draw upon things they’re already learning through the science standards for elementary age children, but they’re putting them to use, and they’re working to save the turtle. Of course they do. It’s an exciting unit, it’s based on a story book that sets the stage for it so you get a lot of your reading and ELA minutes and that sort of thing in it, but then really does bring in science and math as they use the engineering design process.

Rebecca: Sounds really fun.

John: It does.

Stacy: It is a lot of fun [LAUGHTER].

Rebecca: I mean I have to admit I asked that question just because I have a toddler and I was just really curious [LAUGHTER].

Stacy: Talking about the new Wee Engineer, WEE, it’s very cute its for preschool kids.

Rebecca: Yes, yes. Yeah, I want to hear about it. [LAUGHTER]

Stacy: Oh, you really do want to hear about it?

Rebecca: No, I really do.

John: She does [LAUGHTER].

Stacy: The new Wee Engineer units that are coming out are meant for the preschool setting where the teacher introduces the problem…and it’s actually not a teacher, it’s a puppet…and so the puppet comes and introduces and says something like “I want to throw a party for my friends, and I want to make this noise maker really loud, and what do you think of my noisemaker?” …and of course it makes no noise. The puppet then says, “Can you help me?“ …and so the students go through an explore stage, where they explore the materials that are available. A lot of the work at this age focuses on helping students think about how a material is made and how that affects its function. So, they explore different materials and then they get to create their own noisemaker in small groups… and they test it… and then they do it again…and they improve (which is a big part of the engineering design process), until they all have really loud noise makers which they then share with each other and they of course give back to the puppet so the puppet can help throw a good party.

Rebecca: I like that it’s given back to the puppet so that the teachers don’t go crazy. [LAUGHTER]

Stacy: Yes, that would be a critical part of not driving the poor preschool teacher insane.

Rebecca: That sounds like a lot of fun. Maybe I need to go back and teach preschool engineering instead of web design. [LAUGHTER]

John: But, so many students get turned off early on and reaching them early can be really effective in stimulating later interest.

Stacy: They do.

Rebecca: Can you talk a little bit about what you’ve done with STEM for girls and other underrepresented groups and how to get them interested and excited about STEM?

Stacy: Absolutely. My study of the literature shows that a lot of the things that motivate girls also motivates different underrepresented groups, particularly underrepresented minorities/ethnicities, and is often just generally in line with what is good pedagogy, if people actually stopped and thought about that in engineering. I’ll focus on the girls just because that’s been my wheelhouse for the last seven or eight years now. But, a lot of the research shows that girls are interested in helping people or the environment, or something like that. If we can help frame STEM as being something in which you can help people, we will inherently pull a lot more girls into that field. So, that’s kind of one basic way. Often, if I talk to a girl, she’ll say she wants to be a doctor or nurse or something like that in the medical field, because it’s so painfully obvious of how you help people. I try to turn her into thinking about engineering and what engineering really is. Majors like biomedical engineering and environmental engineering are often popular with women, because again, it’s obvious how you’re helping people or things. But if engineers are good, and there’s actually a whole book called Changing the Conversation published by the National Academy of Engineering, on how do we praise engineering appropriately, because it is all about solving human needs and want. If we can present the field more accurately and more fully that will help. I think also when I look at a lot of these things, I like to be very explicit about things like stereotype threat, or implicit bias, or imposter syndrome, and I try to be very overt about teaching students what these things are so that they can recognize it in themselves, know what it is. There’s something about identifying it. I even still have imposter syndrome at times, where I feel like somebody’s gonna figure me out…that I’m not actually that good at engineering education, despite having just been named a fellow of a prestigious Society, I feel like still somebody might figure that out. But I know what it is, I can call it out and say you’re just having a case of imposter syndrome and, somehow, it’s easier to move aside and move along if you know that it’s a real thing and you’re not the only person who has some of these…I call them issues, I’m not sure that’s the right word.

Rebecca: I agree with you. The ability to name it out and file it away allows you to move forward. When I finally learned what some of those things were as a designer, I too, was able to overcome some of those hurdles.

Stacy: I guess the other thing I’ll add, is Carol Dweck’s work with growth mindset, has really put a name to something…about having the ability to think of your brain as a muscle that you can flex and you can grow and it can get stronger. I think letting students know that that’s a thing. Or, at the school where I worked most recently, you were not allowed to say “I’m not good at whatever it was,” you were only allowed to say “I’m not good at _____ yet.” …and I really appreciated that word “yet” there, and the implication that you can and will be good with it, but it’s going to take some hard work, and things don’t always come easily…whether you’re gifted or not doesn’t really matter, you still have to work to accomplish anything good.

John: Besides stereotype threat, implicit bias, and imposter syndrome, what are some of the things that are being done in classes now that deter women and minorities from entering engineering and other STEM fields?

Stacy: The first thing that pops in my mind there is thinking about the examples that are used in a classroom. If there are examples that are supposed to illustrate some concept, yet they are completely unfamiliar to you because the situation in which you’re growing up provides you no context for experiencing that or understanding that, you’re immediately set at a disadvantage in the classroom, and that’s not going to encourage anyone to want to continue in that field. I think there’s also some cases of just downright bias. I had a professor in college that didn’t really seem to think women should be engineers, and well I do know that that is improved, that’s not gone. There are cases of bias that are still out there. I also think a lot about parents and the role of parents, and what they believe their children, their daughters especially, can do…and what’s appropriate for them. Because there are some cultures that have a lot of bias kind of built into them and so it’s about changing the way parents think. Because if a student…if her parents don’t think she should study engineering or science or something, she’s probably not gonna go study that in college. So, we need our parents to understand what these fields are about…educate them…and then get them as a part of our moving more and more diverse people into these STEM fields.

Rebecca: One of the things that I think a lot about is the relationship between design and engineering. As a visual designer, I know that I end up with a lot of students who seem to have a fear of math, or a belief that they just can’t do math, which the process of engineering in the process of visual design is, I don’t know, almost exactly the same. So it’s always interesting to me that they err on the side of the arts thinking that they’re somehow avoiding math, but then of course they discover that there is math there too. Are there things that we can do to help overcome this…I don’t know…. it’s like almost like a preemptive strike, that like “Oh, there’s math. so I obviously can’t participate in this.”

Stacy: I hope so. I feel like we’re making some strides in that area, because you’re right, it is often math that is the big hang-up on why people don’t stay in STEM. Some of that is from having one of your parents, especially the mom, saying “Oh, Honey, I wasn’t good at math, you don’t have to be either.” …which, of course, we would never in a million years say about reading or a lot of other areas. So, I’m not sure why we say that about math sometimes. I think we’ve got to figure out how to let the math come naturally; that, if it’s a part of some problem that you are actually interested in solving, you have empathy for your client, and you’re invested in it, the teachers picked a good problem…“Oh gosh, look we’re gonna have to do this math here” and suddenly it makes sense why you’re doing the math…and you have a reason to want to do it. I think those are critical things that we need to have in our math sequences from elementary on up, so that students don’t develop this hatred or fear of it that is somehow irrational. I also think that while there is math in engineering, not every engineer does mathematics all day long. So, there is some conceptual understandings you have to obtain in order to become an engineer, but it doesn’t mean you sit around and solve differential equations all day long, necessarily. Some can, but many don’t.

Rebecca: I think that’s an important thing. I think there’s a lot of fields where we just assume that people just do math all day and it’s just a misunderstanding or misconception about the field. I think it’s also, sometimes, we present some things in such an abstract way that it doesn’t seem relevant. So, I always like to share with my students the experience that I had around geometry. When I was learning geometry in high school, I could do geometry, I could answer the questions correctly, but I never really understood what the point was and like “I’m never gonna use this.” Then I started doing more programming stuff and made visual interfaces and then all of a sudden was like: “I understand why this is relevant” [LAUGHTER] and I had that breakthrough moment where I was using all kinds of different geometry equations and things to create visual interfaces, essentially.” So, I share that with students and that sometimes helps a little bit. I could put in the math, and then all of a sudden I saw a visual, and then it just clicked and made sense.

Stacy: …made sense…it had purpose to it.

Rebecca: Yeah.

Stacy: I think a lot of math traditionally has been taught like as this separate silo, never to be used…and sometimes I think it’s because the math teachers themselves don’t know when it’s used. They don’t know the science or whatever the other field is…or psychology…and there plenty of places with statistics that use math. But, I feel like we have to lead with those things. So, when I was a high school math teacher and I wanted to teach sinusoids, I would lead with “What’s the temperature gonna be on your graduation day?” …and so we would have to develop this whole model to predict what the temperature was going to be on their high school graduation day – which was not just in a few months, most of them are juniors taking the class. So, we would have to develop this whole mathematical model which involves sinusoids and all the different parameters of one, and then on the test I would give for that unit they had to answer that question for me. It was always fun on their actual graduation day to see how close we had come.
[LAUGHTER]

Rebecca: That sounds like a lot of fun and a great way to make things seem relevant.

Stacy: mm-hmm.

John: So, one thing that would help is if math instruction and science instruction was made more meaningful by using more meaningful examples, so that the math is motivated…so people can get past the fear, because they see that there’s a purpose to it it’s difficult to do that. But, you’re doing some of that with the Nashville schools. I hope we see that more nationwide.
Could you talk a little bit about your work with Engineering is Elementary in designing curriculum?

Stacy: Sure. I have been affiliated with the Engineering is Elementary program for about a decade now. I have followed their work…their research as it was designed and presented at the American Society for Engineering Education. I was super impressed along the way that they were actually doing real education research and they weren’t just developing some curriculum and going “Oh, look how many people use our curriculum.” They were actually looking to see if learning objectives were being met and things like that. So, I’ve had a lot of respect for them. About ten years ago, I affiliated myself with them and became one of their endorsed network providers, such that I could provide their workshops whenever I wanted to around the country, I always really loved and admired how they set up their professional development…how its implemented…that the PD itself is based in research. So, when I was looking for a new educational intellectual pursuit to take on in my career, I approached them and asked if I could come work for them, and to my delight they said “Yes.” So, I started working there part time in January and have enjoyed that. There are separate teams within EIE and I serve on the professional development team right now. So, I’m enjoying working with our extended network of partners, so the people I used to be one of, I now work with and help to take the best of what we’re doing in house in Boston and get that material and the best practices for engineering education out to all these people across the country who can help spread the word and get more kids into engineering. So, I’m really enjoying that piece. I’m gonna be developing some of my first, I’ll call it online PD, but it’ll have some hands-on components to it also, for the adult learners. I think that’s a fun new pursuit for me. In house at EIE, they’ve just created (as I was mentioning earlier) this new Wee Engineer program for preschool and pre-K and there’s also a new EIE for Kindergarten, which I’m thrilled to see, because those age groups desperately need some authentic well-designed, well-researched curriculum. That’s kind of been my role right now. We’ve got some middle school projects as well, but I’m really enjoying starting young and going all the way up through eighth grade and looking at how do you do that best.

Rebecca: As a college faculty member, I’m interested in how you got involved in more of the P-12 things. How might you encourage other faculty, no matter what their discipline is, to get involved at those earlier levels?

Stacy: Great question. I think it’s kind of fun to see, once it clicks to faculty members that what goes on in P-12 definitely affects what can go on at the university level. Some of it can even be slightly self-serving in that they want more students or more diverse students to enter into the university process. So, I think that’s part of it. I think that it’s fun to help a university faculty member see how they can take their passion and enthusiasm that they have for whatever their research field is and take that down to younger kids and distill it to the basics, but increase people’s understanding of their field overall. So, I like that piece of it. There’s the other motivating factor, if you’re gonna apply for like a National Science Foundation grant you have to have an education and outreach component. So, that there’s that external motivator as well to think about how could I be involved in this process. I also worry a lot about we have these new next generation science standards. which I think are quite good but they have a lot of engineering in them. So, who exactly do we think is going to help the K-12 teachers know how to do that, and do it authentically, if we ourselves are not out there helping them and teaching them.

Rebecca: What would you encourage a faculty to do as their first step to get involved in P-12
STACY. I’m trying to think of one single first step…probably, reach out to your kids’ local school and listen…ask the teachers and the administrators…but especially the teachers…ask them what they need. Because the teacher will know. He or she may not know how to go about getting it, but they will know what they need. Don’t go in and be like “I know everything” when in fact you really don’t know everything about what it’s like to be a K-12 teacher…but go in and ask how to be helpful. Listen to what they say and honor the fact that they have to be standards-based.

Rebecca: Sounds like really good advice, but at times, it’s just that little encouragement of what that step could be is helpful. So, thanks for that nugget.

Stacy: Yeah. I mean, go ask. People really want you to.

John: …and in many disciplines, I think, there are organizations that work with elementary and secondary schools. In economics, there’s the centers for economic education spread throughout the country, who do work with middle schools and high schools in providing some educational resources. I don’t know how common that is in other disciplines. Is there anything like that in art?

Rebecca: There is something more general for art, but not for design. Design stuff is kind of under the umbrella of art which, depending on what your position is, you might not think that they’re entirely related. They are, but they aren’t, you know. It’s kind of complicated. {LAUGHTER]

John: You’re working on a new advanced course in engineering for high school students. Could you talk a little bit about that?

Stacy: Yeah, this is a project that has been near and dear to my heart, and some of my wonderful colleagues, particularly from the University of Maryland (Dr. Leigh Abts, in particular). We have been trying to get an AP engineering course started for 14 years now, and when we started, really, the colleges weren’t ready for it. The schools of engineering were not interested in it. They weren’t interested in accepting credit for it. They didn’t really see the value of it. Thankfully, that has changed over time, which I’m really excited about. The College Board got interested in AP engineering and created a framework for the course. They had to put the brakes on that for a little while while they focused on the redesign of the SAT, but now they’re back to being interested in it. So, my team approached the National Science Foundation who said “Yes, we’re on board with this. This needs to happen.” So, we’re modeling our upcoming work off the very successful work of creating the new AP computer science principles course which has been a highly successful course for AP…and it’s also really successful in that a lot of women and underrepresented students are taking the course and taking the test, so that’s super exciting. So, we’re trying to draw some best practices from that and, knock on wood, I hope that the NSF will approve our final proposal to begin the work later this fall. The basic idea around it is we’re going to finish the framework…make sure it’s right…and we’re going to be developing a sample curricula and sample professional development for that. As you may know, with AP courses there isn’t any one set curricula that you’re expected to follow. You can do it however you see fit. You just have to make sure it fits the framework. So, we’ll be developing some sample ones and then we’ll be partnering with high schools. We’re hoping for about 70 high schools all over the country and a lot of diverse settings to train their teachers and have them work with our students and do engineering design at that level. We’re looking at having ultimately an assessment that is a bit like AP art studio actually in that we’re hoping for a portfolio process where you would submit engineering design work that you’ve done over the course of the semester or the year, and then that work would later be evaluated for possible engineering credit.

Rebecca: That sounds like an exciting advancement.

Stacy: I hope so. we’ve had over 110 Deans so far say that they’ll be interested in giving some sort of credit for it. I think it’ll be interesting to see how the universities handle it. I think some might give credit for their actual Intro to Engineering course while others might give it as more general science and engineering credit. I think that the universities now see it as a great way to get more STEM literacy in our population and I think they’ve started to see it as a great way to get more diverse students into their programs, because they will have done engineering at a younger age and done it in the more friendly confines of their local high school.

Rebecca: …and perhaps introduced populations who aren’t really familiar with the field at all to what the field is rather than expecting college students to just magically know what all of our disciplines are.

Stacy: Right, that’s true of a lot of disciplines, so it’s not just engineering.

Rebecca: Yeah, exactly.

Stacy: There aren’t tons of schools that have economics in them but you’re probably not gonna major in it if you don’t know what it is.

John: Actually, I think most schools do now have economics. It’s part of the New York State curriculum and I think most states do have at least one semester economics course. But, it took many many years before that was widespread.

Stacy: We’re trying to catch up with you, John.

John: It’s not always taught by people who know much about economics, unfortunately…. [LAUGHTER] …which is one of the reasons why the Centers do so much work.

Stacy: I would mention that if someone is interested in engineering education, I know that there are now actual programs in engineering education. You can get graduate degrees in it, and I would also steer them towards the American Society for Engineering Education. It’s a wonderful Society. It is the place to go to for what the best pedagogy is in engineering education, and to find your people there. They have lots of divisions, some are specific to your field of engineering and then we have a wonderful pre-college division there as well. So, it can be a great resource if anybody who’s listening wants to jump in jump in and join it.

Rebecca: So, we generally wrap up our podcast by asking what’s next?

Stacy: What’s next? Well, I feel like I’ve hit some of the “what’s next” because I’m in this great transition period in my career, which I’m excited about. So, I’m hoping that what’s next is that personally I’m able to view engineering education from preschool all the way through 12th grade and then into college and look for “How does that pathway work?” Are there things that are missing? Are there things we should be doing differently? So, I’m excited about taking that long view across engineering education and I’m always looking for new collaborators and people who are as excited about the field as I am.

Rebecca: Thank you so much for joining us. I hope that you’ve motivated a lot of others to think about their own disciplines plans from the elementary level all the way up through the university level. Sometimes, that longitudinal perspective can really help us have better perspective on what we’re teaching in higher ed.

Stacy: Definitely. Just to think about like “What matters? “What’s actually important?” It’s easy to get caught up in the minutiae of these little things you have to be sure you’ve mentioned to your students. Not really. Focus in on the big thing.

John: …and if you really want to do something about the gender imbalance in STEM fields you do have to reach out earlier because by the time you get to high school, people have already been sorted out. So, it’s really important to do that sort of work early.

Stacy: Very true. Most of the girls are getting sorted out late elementary to middle school, at the latest. So, you’re absolutely right there.

John: Well, thank you.

Stacy: Well, thanks for having me. This was fun. I appreciate you reaching out to me John, I was flattered.

John: Thank you, Stacy. We very much appreciate you joining us today.
[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Theme music by Michael Gary Brewer.

44. Industry realistic experiences

Student motivation is enhanced when students see that the work they are doing is relevant to their future careers. In this episode, Dr. Bastian Tenbergen, an assistant professor of Computer Science at the State University of New York at Oswego, joins us to discuss how industry realistic projects may be used to enhance learning in software engineering classes.

Show Notes

  • Daun, M., Salmon, A., Tenbergen, B., Weyer, T., & Pohl, K. (2014, April). Industrial case studies in graduate requirements engineering courses: The impact on student motivation. In Software Engineering Education and Training (CSEE&T), 2014 IEEE 27th Conference on (pp. 3-12). IEEE.
  • Daun, M., Salmon, A., Weyer, T., Pohl, K., & Tenbergen, B. (2016, April). Project-based learning with examples from industry in university courses: an experience report from an undergraduate requirements engineering course. In Software Engineering Education and Training (CSEET), 2016 IEEE 29th International Conference on (pp. 184-193). IEEE.
  • Dijkstra, E. W. (1959). “A Note on Two Problems in Connection with Graphs.” Numerische Math. 1, 269-271.

Transcript

John: Student motivation is enhanced when students see that the work they’re doing in their classes is relevant to their future careers. In this episode we examine how industry realistic projects may be used to enhance learning in software engineering classes.
Thanks for joining us for “Tea for Teaching,” an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego [Music]

John: Today our guest is Dr. Bastian Tenbergen, an assistant professor of computer science at the State University of New York at Oswego. Welcome!

Bastian: Thank you, thanks for having me.

Rebecca: Today our teas are…

Bastian: Well, upon John’s recommendation, I’m having the mint herbal mix tea, which is excellent! I’m a peppermint tea drinker, so this is blowing my mind right now.

Rebecca: Excellent!

John: I’m having ginger tea.

Rebecca: I’m having Prince of Wales today.

Bastian: I like the ginger tea, that is my favorite tea.

John: It’s good.

Bastian: Ginger and fennel and peppermint, those are my three.

John: We invited you here to talk a bit about the projects that you have students do in your computer science classes. What classes do you generally teach?

Bastian: I’m teaching in the computer science department, but I’m mostly teaching software engineering courses. We actually have two separate majors: we have computer science majors (Bachelor of Arts and Bachelor of Science) and we also have a software engineering Bachelor of Science program. People usually confuse software engineering and computer science or at the very least don’t really know what the differentiation is. In contrast to computer science where it’s really all about programming and all about finding optimal algorithms to solve problem x for person Y, software engineering is concerned with the process of development from A to Z. So from requirements all the way to programming which is a small part of it, all the way to Quality Assurance and also budgeting. Also, the business aspect of it, so it has a wider focus.

Rebecca: It’s a little more client facing?

Bastian: Very much client facing, yes. By trade I’m a requirements engineer you can say and a very smart person who very recently submitted his PhD dissertation (which I’m very proud of him that he did finally did that). He wants to find requirements engineering as a socio technical process that implements the vision of a system given the time and budget constraints that you have. They usually also call us the context of the system, the developmental context of a system. It’s the budget, the time, the resources you have and such things. Those are considerations during software engineering.

John: In what classes do you have students engage in projects?

Bastian: Well it is very hard to teach computer science without actually using projects. You can teach the skills but at some point the art of making software becomes more than the alignment of skills in a particular way. Legitimately almost all classes we teach have a very heavy focus on projects. I’m teaching a software and safety requirements engineering course which is project-based, at least a quarter to half the students grades depends on the project. I’m also teaching a software quality assurance class where at least a quarter, sometimes half of the grade depends on project performance. I’m also teaching occasionally capstone courses, where the capstone experience in the software engineering program really tries to simulate how an independent developer develops a spoke software for one individual client and one of my favorite things to teach is a class called “Software Design”. The term design implies software architecture but it’s not just that. For those software engineers out there listening, this particular course is called that for historic reasons, but it’s really a design process class. The entire class collaborates together on producing one substantial piece of software, which is usually on the first day of class. I demand like big evil Papa Smurf that this project could be marketable, so the explicit goal is we want to market it, which sometimes works and sometimes doesn’t, but that’s the goal. Then we differentiate the students into teams and have a database team, a GUI team, we have graduate students at our university that specifically focus on usability and human factors so we have those as a team, we have requirements teams, we have Quality Assurance teams. They have to learn not only how to work together, they also have to learn how to apply their skills, have to learn how to best make design decisions, how to communicate them and not only how to communicate them with like-minded peers that are also scientifically or engineering capable but also with a stakeholder. Software engineering in general is very focused on the people who are giving the money for a project. In my classes I really focus on the fact that students should be able to argue their rationales, not to other engineers and not to other technicians, but to their grandmother because if you can explain it to your grandma, you can explain it to the person who gives you money in the project; and that usually worked well.

John: How early in the term do students decide on the project?

Bastian: So, It depends. It depends on the course. In my requirements engineering and software quality assurance class where we also teach skills, we also teach requirements, solicitation, or you teach let’s say data flow based testing, which is a new technique for them to pick up. There, I usually pick the projects for them or if they have a particular good idea we’ll discuss it, but usually it’s in the first week or so that they finalize the project. In capstone classes and in the software design process class, I usually conceive the project ideas and then we make the necessary choices, let’s say the necessary preliminary choices in the first week. What I mean by necessary and preliminary choices it’s this; I basically say “I want a universal all-transfunctionater” and no one has any clue what that is and I say “great it’s your job to ask the stakeholder, who is also me, what I mean by that.” Then the requirements team would differentiate the people into teams and the people who self-select into requirements they say: “Ok, well Bastian, what did you mean by that?” …and I say “Well, I meant… really… whatever… a cow milking device.” So the project kind of takes shape. So, I force them to come up with the requirements and to get them out of me, so that, as an instructor I basically have a dual role… or actually triple role, sometimes quadruple role and I’m project manager for them. I’m also the stakeholder, I’m also the person who gives them advice and the instructor that says “dude you shouldn’t do this because X & Y & Z or whatever. Or, maybe here’s a great idea that someone else just had and maybe try this.” More often than not I’m also the conflict solver and a psychologist that lets them cry on the shoulder because at some point during the semester everyone is just frustrated. This is part of the experience I guess but that’s why I usually tell my students the trick is to be successful despite other humans and once that idea clicks, working together never becomes a problem ever again. So as you lose one conflict earlier in the semester and then it kind of dissolves and this is when you see the students go from students to professionals. It’s my favorite class to teach because you can see how the students go from “professor, how do you want this” to “well Bastian I know you said you want a cow milking device but see we don’t have any cows, so how about we build you this instead”. It’s important in these kinds of projects for them to be able to communicate what the stakeholder wanted versus what we can conceivably give to the stakeholder given the time and the budget and the people that we have on staff.

Rebecca: Or what this stakeholder may actually need and doesn’t realize that they need.

Bastian: That’s right! Two years ago, I co-taught to this class for the first time which was great because then we could literally play good-cop and bad-cop. One stakeholder and one instructor will always be against the ideas, which believe it or not wasn’t necessarily me, and the other one was always in favor and would always say “oh yes that’s fine, that’s fine, Keep going”. But you know even if you have someone who constantly approves of what you do you don’t know whether or not you’re actually making any good progress. So it may feel good to have your ego stroked and be told that yes everything is great but that doesn’t necessarily mean that you’re making useful progress. Really in the end the only way you learn is if you make mistakes. On the other hand of course being told everything is bad or everything is completely horrible and how dare you even propose this doesn’t help either. So the truth is somewhere in the middle and it’s for the students to find out what goes. That’s the tricky part about teaching this kind of class, is to guide the knowledge discovery process such that they find it but they can still be successful despite having to do all the work themselves really.

Rebecca: So you’re describing mostly the setup for your software design class right? Which is a big team right that has small teams on it, but you’re all working on the same project.

Bastian: Yes.

Rebecca: Are your other projects and your other classes also set up so that everyone’s working on the same project or individuals working on a project? How are the setup similar are different?

Bastian: You have teams of students I have a very much focused on that that students would at least together with one other person. And the reason is, four eyes see more than two eyes, that’s why. Plus I encourage them like, hey, you know if you talk to another person, if you vocalize your problems, it helps, it stimulates your thinking. So that’s why I do this for example my requirements class, I give the general theme of the project and then let the students do some of it on their own. For example, a little while ago when I taught this software and safety the requirements class first here in the US, I gave the students the opportunity to I said, “okay, we have these cyber physical Rovers or robots, never mind what cyber physical systems are but it’s a buzzword and they can do certain things something makes them special”. We discussed this in class and I said, “we have these robots, and I want you to do something cool with them.” “They each have individual functionalities, pick one for different sensors, different robots had different sensors, pick one and do something fun with it”. And they pitched the project ideas. For example, one of them said, “I want my robot to exit a maze.” Great idea do it. Another person said, “I want my robot to use the camera and use computer vision to recognize another robot and drive after him”. And it was a cool project. Another team of thing was three students actually said, “no we don’t like the robots we’d much rather do something else and here’s an idea”, and I said “okay”. Soon as the learning objectives that I have to find in my syllabus are roughly aligned, I’ll let them go. My general philosophy is if the student has a better idea than me and can argue it, ok. Because I want to learn something too, right? (laughter). So I let them do it and let him explore it if they have the idea right.

John: The students would have more ownership till when they come up with the idea.

Bastian: True. Usually I’m not sure if it’s me over the project or it’s just those cute little robots that we have, but usually students are quite enthusiastic about projects. For the coming semester believe it or not we bought programmable slot cars. Remember those slot cars that you used to race on the like little tracks, you a little controller in your hand you can push more and less gas and throttle. We bought programmable ones and we’re gonna be using that in a project. I’m super excited about this and can’t wait to play with that. I’m hoping students will be excited about this too. And if they’re not then fine they’re not expensive.(Laughter). Plus we have several other faculty in our department who are quite excited about these. I’m not going to tell you the name of the manufacturer but they have a very cool API, which is an application programming interface, which is really simple and open. I haven’t tried them out yet, so I’m hoping it that’s a needle platform to automotive software engineering projects which would be cool.

Rebecca: So, as your students are working in teams and you’re trying to make sure that they’re prepared for professional life, right? You’ve talked a little bit about thinking about clients and things like that. How do you make sure that the problem that they’re solving is realistic and it’s not pared down so much that it’s unrealistic? Sometimes when students self define a project, it’s in a context that wouldn’t generally exist when they are working on their own unless they’re at a startup.

Bastian: That is so true. I would argue that finding the project not necessarily the scope, but the project domain is probably one of the two hardest things about doing the project. In fact, I’m not sure if I’m allowed to say this and make some advertisement on my own behalf here, but colleagues of mine and I wrote two academic papers and we’ve just submitted the third one on project-based learning in industry-realistic case examples in software engineering to a fairly substantial fairly high ranking conference. The industry realistic examples, they usually reflect one or two aspects that you would commonly find in let’s say industrial development projects. For example, the problem of, let’s say sensor integration. If you have a little robot and you tell the robot to rotate 90 degrees, you can know whether or not that thing actually turned 90 degrees because the one motor if you have two wheels, assuming you have a two-wheeled robot one motor might be have different manufacturing tolerances and maybe a little bit stronger than the other one, so you may be turned 89 degrees, maybe you turn 94 degrees. So how do you fix that? Well you could put a little sensor on it that does that, but the only rotational sensors you have they are going to be inaccurate too. Especially if you have let’s say have the robot run on carpet rather than tile. All of a sudden the physical setting and that the robot is in has a great impact on the software that you’re developing, and that is an industry realistic problem. Let’s say you fly an autonomous aerial vehicle somewhere and try to detect wildfires, which we are currently experiencing a very hot summer with a lot of drought. So they do this, they use drones to detect wildfires. How do you know you’re actually currently flying through smoke as opposed to through humidity or through fog or through a regular cloud? You have to use sensors. It’s a realistic problem. So the domain flying an actual drone is hard, so we use a little robot which however has the same kind of problems. I was very fortunate that earlier in life, I was working with some industrial companies in research projects and so it’s relatively easy for me to figure out what could be a challenge that the software developer or software engineers is going to be facing. So in those two papers that are just described, we focused on how to apply industry realistic case examples and we figured out what kind of properties these have. For example, you want to be sure that the project that you give to your students doesn’t have a bunch of challenges, but just one is usually enough, just to focus on one little challenge. For example, get the little robot to rotate accurately, but you don’t tell them make a project that lets the robot rotate, because that’s boring. Instead you say, “hey, why don’t you write an overtaking algorithm for robots?” And usually you know full well that in order to make those robots actually overtake one another like cars on a highway, a lot of things have to fall into place. First for example, you have to figure out how to make this robot drive straight and that is already a project in an art of itself. So the other important criterion for these industry realistic projects is to have the project scalable. So toward the end of the semester I usually joke with my students and say, “well, if you can’t finish your project in time, it’s either because you didn’t scope the requirements right, or because you bit off more you can chew, development is harder than you initially thought, or maybe because we haven’t redefined success yet.” So if you can’t be successful redefine success. Which when I say that really what I mean is I tell them, listen, you can’t deliver what you wanted to deliver, fine, not a problem happens all the time in reality, instead tell me what we can expect. Given the time that’s left what can we expect. “Well, we can actually make the robot overtake”, they will say, “but we can make it drive straight with a certain level of accuracy.” That seems boring and uninteresting when I say it like this but it’s actually a remarkable feat. At the end of the semester, two kinds of students those that are happy to be done because this was horrible experience, the minority thankfully, or you have the people that say, “oh my god, had no idea how hard it actually is to interface hardware and software.”

Rebecca: Really a big lesson in scoping, it’s like how do you break a big project into small pieces.

Bastian: Absolutely.

Rebecca: Understand that small pieces have to be completed before you can put them together to make a big piece. It’s like modular design.

Bastian: Yeah, absolutely. Modular design is one of those keywords buzzwords almost from the 90s, but they were right. You divide and conquer is a recurring theme in computer science that works everywhere. If you want to sort numbers you divide and you conquer it’s the fastest way to do it and if you want to develop a software project you divide and you conquer. Your first build project one and project two. You can scope this whatever way you want. Very often actually I have students who halfway through the project realize the potential that the project has that they’re working on and say, “hey Bastion, I really would like to bring this project into this direction instead I know you said overtake algorithm, but let’s do a path finding algorithm instead.” Esker Dijkstra in the 1960s wrote basically the silver bullet of shortest path algorithms and, can I implement that and put it in the robot? And why not? Just last semester I had someone interested in that doing it. The third characteristic about these projects is don’t be a stickler too much for what the industry really experiences and let the student figure it out on their own. And the one hand you could simulate what companies develop software to particular degree. So you could say, oh we are all now going to fill out application slips or vacation slips or things like this right, but that this misleading from the art of developing software. On the other hand when you tell the student hey listen or when the student asks, “hey listen, I want to bring this in another direction because I find this really interesting,” usually what comes out is something really rewarding, In my experience at least. So the third concept is don’t overdo it students will by themselves, with enough enthusiasm, drive it into a direction that is going to blow your mind, theirs and yours.

Rebecca: So when students are working together in teams and they’re taking on kind of different roles. How do you help the students divide those rules but then also make sure that they’re learning all of the skills or techniques that you want them to learn.

Bastian: That’s hard it’s really really hard and I would say that there’s no silver bullet of how to do this. It is an unfortunate truth that the larger the project is the more people are working on the same project, the higher the chance that at least one person is simply left out and you can be the kind of person that says okay, let’s try to live this person up to make sure that they learn something, but to be entirely honest, in part, in my opinion it’s a component of the experience to make yourself available to your team. So what I do throughout the semester is encourage students to contribute any way they can and students miss understand sometimes from a grading perspective that contributing means being the natural-born leader. In my experience, every team no matter what has one or maybe two people who are really great at the technology and also really great with people and their form naturally adopt the role of the leader. Assigning a leader doesn’t really work all that often. You can say okay you’re a graduate student so you’ll have more management responsibilities and that usually works. But often there’s one non graduate student who’s also fulfilling this managerial role so part of the experience is to find any way you can possibly be helpful for your team this doesn’t necessarily have to be the leader role. You cannot be a leader and be a rather shy quiet person and still get an A in project based courses, the way I teach him. Simply because what does an A mean? An A means here an excellent outstanding student and when are you excellent outstanding student? Well, in these cases when you’re an excellent outstanding team member for your team. When are you doing that? Well, when you contribute stuff any way you can to your team such that your team can continue. I’ll give you an example, if you are the kind of person that never volunteers presentations in class, that never contacts me as the instructor with questions, that never has an management important role in the team but manages all the background communication, implements all the code, and does all the right things in the simply couldn’t contribute couldn’t do what they’re supposed to do if it wasn’t for your input; you’re an A student, regardless of whether or not you’re very outspoken and outgoing or not. On the other hand, if you are a student who talks a lot and who is volunteering a lot, and who is putting themselves in the limelight a lot, but at the end of the day your team can’t count on you because you didn’t show up for the team meeting or because you promised something but never delivered or because the stuff that you deliver is of poor quality and your team decides to drop it and not use your work. Then you’re clearly on the other end of the grading spectrum. So I have a rubric, a rubric system where I say oh can a student clearly is the backbone of the team any way possible a B student is delivering useful stuff in regular intervals and C student is well useful when being assigned work, right, and a D student is unfortunately not useful even when prompted and an e student is the kind of student where the team said listen we’ve asked you 15 times you haven’t done a darn thing we’re done with you.

John: We should know that as we go for some reason we use E’s instead of F’s.

Bastian: Oh that’s right. I’m sorry.

Rebecca: Its alphabetical.

John: It doesn’t make sense to any of us but it’s been done here for a long time.

Bastian: It’s true. So a student that is failing the course with an E or other universities with an F usually those students know that they are. Usually before they are even assigned a failing grade I’ve had numerous conversations with them not as the manager, not asked stakeholder, but as the Papa Smurf (laughter) who says listen, if you want to pass this class, and for software engineering students in our university this class is a core requirement, so they have to have a passing grade in this class to graduate. I say listen, right now you’re not. We’re also doing peer evaluations so some people could say well if you were the one that subjectively evaluates the students isn’t that unfair and the answer is yes, of course. So I’ve experimented with this, just evaluations by me, and I had some good experiences with it and also some very bad ones, unfortunately. So within disputes, and it happens occasionally. What I like to do is peer evaluations where students within the same team evaluate other team members on a scale of say 25 points and usually, and remarkably, these peer evaluations match my subjective opinion almost all the time, 100 %. Students when they evaluate others are usually little positively biased and they are reluctant to evaluate people really badly, but if you ignore that, the subjective evaluation students have of each other are matching my observations very well.

John: How often do they get feedback in terms of how well they’re doing?

Bastian: Every day, every day. We meet usually in this class, we are meeting three times a week or the university has allotted three meeting times a week. I like to schedule two meetings where I’m there and they are reporting to me in daily scrums, those of you who are software engineers,yes we’re doing AGILE methodology specifically scrum. We do daily scrum so it’s basically, you stand up when and you say this is what we have done from last time until today, this is what we’re currently working on, this is what we’ll do next, these are the roadblocks, these are possible problems, and these are questions that we have. Five minutes, everyone does it and usually takes the entire class period to figure out problems, to resolve roadblocks, and most of the time it’s minor things but gotta get done because it’s the planning for the rest. So, during that is when I provide feedback by saying hey have you done this yet or have you thought about that yet, or John Doe here, was supposed to deliver this and that, did they? On the other hand, I’ve very often we have experiences that students say well, see our friend Jane Doe here foresaw two weeks ago that this is going to be a problem, so she already did this and that in anticipation. That’s how you know you have a really great student at hand, right, when they can anticipate problems in the future but would usually only experienced engineers are able to do. So they get feedback every time. What I do however, is the third class meeting that we have, I usually reserve for project work. Because that is the one day in the academic schedule for all students in the class, and if you have 30 people in the class, that I know they have time. Especially at the beginning of the semester I often hear things like, oh we don’t have class on Friday. I’m like, no, no, no, no no, you have class. I might not be there and the reality is that of course I’m there, I’m just then the next room letting them duke it out, and when the shouting or the crying gets too loud, I walk in. Or they decide on things and they have a question and needed it answered right then right there, so they walk over to the other room, or wherever, I am and they ask me. Or I just sit quietly in the room and let the students plan the work on their own. So, the idea is that the third meeting of the week is usually when they get to make progress when they need other people to be present. We also usually coordinate using online chat functions, we’ve used Discord.

John: This is used in a lot of gaming.

Bastian: Yes yes I use them gaming a lot right? Plus all my students they’re all familiar with it because they’re usually all gamers. And we even have a little Steam community going because, you’re nerds like that. So they coordinate through Discord and sometimes they say, hey Bastian is a fine if we don’t meet in person because John and Jane are out of town because, whatever, wedding or sick or whatever, is it okay if we do this online? I say sure, I don’t care how you get it done, just get it done. That’s all I care about. I care about you make progress any way you can. Next semester I’m actually preparing for having this class for the first time in a sort of hybrid fashion. Hybrid in how a university means a portion of the course is online the other portion is a physical in class meetings and what I want to experiment with is, moving this course to an entirely online fashion. Basically simulate how offshore development works. Let’s say you have a team working in Atlanta, you have a team that works here in upstate New York, and you have another team in India or Poland or Germany, and they work together they have to coordinate somehow. So we’re gonna do this next semester. I’m excited, really excited for that.

John: Interesting. Will there be a synchronous component where you have everyone report?

John: Absolutely. So the reason why I said hybrid is because we’re gonna meet exactly twice in person. It’s going to be at the first class we’re going to actually physically meet. I tell them that from now on we’re not going to meet anymore. Instead, we’re going to meet online using an online meeting tool. The university has a couple of licenses that we’re friendly enough to allow me to use one. So we’re using this tool, we’re doing online meetings where everyone has to be present and has to do the same things we would otherwise do if we had physical, in-class meetings; the daily scrum, this is what I’ve been working on, this is what I’m gonna do next, this is what we as a team have been doing. So we still have the immediate feedback component, we can still plan ahead and we can still do all of this. The second time we meet will be at the end of the semester when we present the final project and when we show the final implementation to the stakeholder. Basically like a sales pitch. Of course that’s gonna be problematic because specially the usability folks, those part of the team who are going to be conducting actual usability tests with human subjects committee approval and everything, so we do it the actual way that a company does it, they of course have to meet. This is for next semester I’m actually thinking about having them fill out mock travel requests just to get them accustomed to this. So we’ll see how this work. I’m quite excited about this prospect. I looked at the class roster the other day and I think I have a really cool crew of really capable people and as things gonna be great.

Rebecca: What are some challenges that you’ve run across teaching project-based classes and some advice that maybe you could give to a faculty who’s newer to this methodology?

Bastian: I would still consider myself new to this. I’m actually junior faculty so I’m only, in quotation marks, an assistant professor at this university for just about three years. But our department usually have four as project involve classes taught by more senior faculty. One of the most significant challenges that have experienced this when you have disruptive students. Every once in a while you have a student who completely hates the idea of projects and frankly I was one of them when I was in grad school, I was I was one of them because at the end of grad school I was like if I hear the word project one more time I’m going to flip out. These days I have a different opinion of this. I understand that some people are just fed up with it and I understand. Especially when they have to work with other people that they don’t know that don’t have the same work ethic that they do, they get frustrated a lot. So a recurring problem is student frustration with other students. That’s why I joke with them and say well this class is not about skill acquisition, I don’t need you to know how to compile code, at this point I expect you know how to do this. I need you to learn how to be successful despite other people in your group. You need to be successful despite the fact that you’re running out of time. That kind of stuff. So it takes a little bit of convincing sometimes but usually you’ll find the trick is to find an amicable solution. Then if there’s conflict between people then talk to both sides and say listen, I’m not your enemy, I’m not here to point fingers, I’m not here to agree with you or disagree with you, I’m here to help you facilitate a compromise. That is sometimes challenging. It happens every single semester, but it’s challenging. My strategy usually is to listen to both sides and say okay and maybe you just used the word, the wrong words, maybe you use the wrong language, maybe there’s cultural differences, you have students from other countries and they might not have the same work ethic that you do they may work 24/7 it feels like and you will really appreciate your weekends off. That is fine that is a fine, thing to do we just need to be upfront about it we just say, listen Jane, I’m not gonna work Sunday nights because Sunday night’s is when I relax. Or hey, I’m sorry Wayne, tomorrow morning 8 o’clock is the only time we can meet, can you somehow make it happen? So it’s really about compromise and it’s the case-by-case thing but my strategy is listen to them all and if they can’t make a decision on their own, then I make one, and they just have to abide by it. Usually it’s not a problem.

John: Which is also a useful job skill because they’re going to be in these environments.

Bastian: Exactly. In fact, when I say we simulate the way a software company develops software, I’m not joking. We really do it. These conflicts that you have in a class like this are literally the same. Most students really appreciate the experience, they may hate going through it but they usually love it at the end. In fact two years ago, I had a graduate student who was a graduate student of human-computer interaction, of which our University has a master’s program, but her background I believe it was art. She came from an art background.

Rebecca: Probably a graphic design student.

Bastian: Um, I’m not certain about that, but probably. The strength of the HCI graduate program is that it has so many people from so many different backgrounds, which is a great asset, and you can draw from really greatly talented people. Unfortunately, the downside is well these people they may have taken exactly one computer science class ,namely introduction to programming, and they have never done anything software, ever, ever again. But this person she hated going through this class she hated every single second of it but now she is working for a rather renown company here in upstate New York and she says I’m really experiencing this every day of my life, and I’m so thankful we went through this. This is the best worst class you’ll ever take in your entire life. It’s not about making students suffer of course it is about making them experience something in a realistic fashion, and tone it down a little bit. I don’t want to be the evil boss, I don’t want to be the guy who okay’s everything, and the truth is somewhere in the middle and usually that kind of pans out. Another really challenging thing though is when you have the disruptive student. Not just someone who’s fed up with projects or fed up with people in the project but actually tries to sabotage it. Not too long ago I had a student who was let’s say, extremely convinced of their own opinion, and this person, they were very sure of their own abilities. They were very keen on arguing they would argue everything until you’re blue in the face. They would misinterpret people stopping to argue because they just fed up with it, with oh they just conceded, I won the argument. So I had this person actually say, what everyone is praising me for my great ideas. I said well, sure, but you’ve done these three components that you’ve developed for this project, and your team has used none of them. Your team is no longer inviting you to team meetings, on my recommendation, because whenever you were at a team meeting they would not get anything done. So what do you think, what do you think this is, this is not okay, this isn’t an okay behavior. So in the end we found a way to help this person become useful after all, for the team, but it was very very challenging. In this particular semester I would think that unfortunately half of my teaching load was probably just taking care of this one person. Later I found out from other faculty that they were difficult in other classes also, so it wasn’t really me or the class, it was just personal issue. Even though this person took a lot of my time, ordinarily this class is the easiest to teach because, I don’t need to prepare anything, I have no preparation some grading afterwards but no preparation. On the other hand, you also have to be ready to face anything. You walk in a classroom and you don’t know what fresh hell awaits you that morning in terms of conflicts, but as I said, it’s only experienced as conflict while you were in it, afterwards you’re laughing it off and everyone is usually happy that it happened this way. So that’s what I’m saying is like a rewarding class to teach, but it’s kind of tough.

Rebecca: I imagine you probably have busy office hours as well with project based learning.

Bastian: Oh yeah. So much so that my faculty website says, office hours by appointment only. In reality it means, if I’m in, I’ll probably have time for you. Because with classes like these problems emerge right then and there, and I don’t mean interpersonal problems I mean, oh snap, we really need to use this one server but the server just went down. What do we do now? Or, we’re using this Google API and Google did what Google loves to do, namely change their API, what do we do now? Or, not too long ago, we were developing Facebook integration and Facebook from one day to the next took away the ability to post across pages on Facebook. So the project was kind of dead in the water, what do you do now? And that’s the problem that emerges immediately and you have to fix it, the students can’t fix it. When the resources that they need vanish, they can’t help themselves, there’s no way they can recover on their own. So that’s when after a short brief moment of panic, where I panic myself, we have to fix it somehow.

Rebecca: And you become the magic wand. [laughter] That’s what my students think when they’re standing in line for project-based learning. It’s like they come in it’s like, please I can’t move forward.

John: Those are all realistic type problems that they will be facing.

Bastian: It happens all the time happens to companies all the time, if you’re in the reality of the situation is Facebook doesn’t just take this away neither does Google. Google as opposed to, for example Oracle, they don’t really change their Java API all that much and if they do they have support for the things that you use to use,it call it deprecated, Google just switches it off. But they don’t do it from one day to the next there is usually a period where they tell you, oh by the way in a year or so we’re gonna switch over this in that server. So technically as a student you could be prepared if you did enough research but realistically, they have to complete this project, and our semesters are 15 weeks long, they have to complete this in 15 weeks so you have to make some concessions. Then we’ll just redefine the scope we just focus on something else. For example, a little while ago Google took away the opportunity of making your own google map, and when I say that is not a google map of let’s say, I don’t know, Oswego New York. Using the Google map engine, make a map of your bedroom, that’s what I meant. So they took away that opportunity or they took away certain functionalities that we wanted in one of our robot projects. I said well, they can’t do that so what I’m gonna do instead? One student suggested, hey, can we use the Unity engine to model a room that robot moves in? I said sure. Unity is a game engine to make video games. I said okay sure, you can do that, but I don’t know unity very well. Actually, I don’t know it at all. So, we have people here on campus who do know this, but I’m having a feeling to become good enough at unity to make this project work we’ll take another semester of itself. So why don’t you do it the easy way? Take a picture of the room that you want to use, and then “restorize” it and just fake it till you make it. So in the end the project was successful despite Google’s API being on.

John: What are some examples of specific topics that are used in design class?

Bastian: So in the software design process class, the first time I taught it here in Oswego, we did a family tree website, like those find your ancestry websites that you can find on the internet. Mainly because my Dad, he now passed away, but my dad was really into that and he wanted a website just to show our own family tree. We did that which was marginally successful. It was a decent family tree some of the features that we initially shot for were not delivered but, you know, we can safely say it was a family tree. A year after that we did an automated clicker system and I know that John here, is very much a proponent of using clickers and classrooms. If you have seen that millionaire quiz show on TV, they have little devices, and you can basically poll the audience in the classroom or in a question or multiple choice type answers. So we implemented it, and I’m of the firm opinion that no student should have to pay money or anything because tuition is already high enough, so we implemented a free one. That was using students own cell phones and wireless network they could poll.

John: You had some classes actually use it as clients for protocall.

Bastian: That’s right. So I used it in my own introduction to programming class. I used that semester, I used them as guinea pigs. They were excited beyond belief. They kind of liked it. It was very buggy of course mainly because doing it over wireless is really bad protocol. Plus if you have a wireless network in a large lecture hall it is an even worse protocol. So there were some problems with it that we couldn’t just solve, that were just unsolvable to us. But in principle, in a small enough audience, let’s say inside of 20 students, it would work great. Last semester was particularly exciting due to a scheduling error by, I’m not gonna say whom, but say by certain administrative forces, I unfortunately and accidentally had twice as many students in this class as I was supposed to. I like to teach this class with like between 15 and let’s say 25. Because we have a lot of students sometimes we have to unfortunately have 30 students in this class. Last semester I had 50, so yeah.

Rebecca: Oops.

Bastian: That was awful. But I decided after I talked to our department head, Doug Lea, and he says well, what you’re gonna do, pick up people and kick them out? We decided that this is a really evil thing to do to students so we just bit in the sour apple and said okay fine, let’s do a red team blue team approach. Where we had the same project and we split the class in half saying you’re team blue, you’re choosing a different design solution than team red. They both implemented a Scrabble clone. Those of you have played Scrabble board game, and we can use words and play words, and the idea is that people would walk by a kiosk system, which is actually running right now and the entrance of our science building here, is a computer in a display case. It’s running a cloned version of Scrabble. People can walk by with their cell phones connect to a little wireless that is emitted there and then they get a hand dealt on their cell phone, then they can play words. Of course they’d have the usual problems like, the first person that walks by plays an unspeakable word, so we made it Oswego themed and say if you play certain words you would get bonuses and such things. I would just mentioned in the coming semester I’m going to teach this class for the first time mainly online and I’m thinking about doing a Productivity type software. Something like it connects to your email account and looks for what your emails are actually about; how much time do you spend in your emails, how much time do you waste? For me, as faculty I always feel like I’m doing 5 % teaching, 3 % research, and 97 million % of miscellaneous administrative stuff, so mostly probably emails.

Rebecca: Mostly email. [laughter]

Bastian: I want to know if that’s true. I want to see what do my email say I am communicating about the most? On the one hand you have to connect to Google’s IMAP account and download emails and then you’d have to some natural language processing to parts of speech in the email and so on. Of course there gonna be privacy issues with this. These days everyone is really concerned about privacy, as they should, so we’re gonna have a little team that is gonna be specifically concerned with making sure that we abide by ISO 27000 privacy regulations. Unless the students have a better idea of course. [laughter]

John: So our last question is, what are you going to do next?

Bastian: I’m really excited. I had a student, I was successful in obtaining funding for a student project over the summer, and this student built an indoor GPS navigation system for robots. Now when I say that I mean mainly the API. So from this grant money we bought a little ultrasonic location beacons, you could say, which can be distributed around the room and the robot gets another location beacon slapped on top of itself, and then the robot knows in relationship to all the other beacons, where it is. Using this little system he implemented a GPS type API that allows us to say, robot go exactly there, and the robot will drive up to two centimeters precise to that position. The robot has obstacle avoidance, it has pathfinding capabilities, and all that stuff. So one of the things that I want to do next is have a fleet of those robots, we have several of those robots, but only one of them is location aware right now. When I put location awareness on several other robots and then simulate let’s say exoplanet exploration, using those little things. Let’s say you have three or four or five or 20 of those robots roaming around in a large room and one of them finds an obstacle and says, hey guys, here’s an obstacle don’t run into. It tells all the other robots where that obstacle is and then the next time when the next robot comes around, to a similar location, and says oh here’s an obstacle, here’s the question; is it the same obstacle? Because if it is, then we don’t have to put two obstacles on the conceptual map, we have to do just one. So it’s something I want to do it also ties into into my research. Like one of the things that I’m really, really focusing on is to make sure that the students just don’t do boring little projects. Every student in computer science has implemented a library system or an ATM, you know boring, been done before. I’ve worked, as I said earlier, in cyber-physical systems and safety-critical requirements and such things, so I use those ideas in my classes and I want them to solve tiny little projects therein. I just mentioned earlier, we bought these programmable slot cars. What I want to do next is do obstacle avoidance and automatic cruise controls with those slot cars and just automotive type software engineering projects. That’s what’s happening. I’m really excited about that too.

Rebecca: Great. Thanks for joining us today.

Bastian: Thank you for having me, I’ve really enjoyed being here.

John: You’re doing some really interesting things there.

Bastian: I’m not doing any of them. [laughter] The students are doing them. I’m just there for the ride, really. [Music]

John: If you’ve enjoyed this podcast please subscribe and leave review on iTunes or your favorite podcast service. To continue the conversation join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes transcripts and other materials on Teaforteaching.com. Music by Michael Gary Brewer. [Music]

32. The Three Little Pigs

What do the three little pigs, the big bad wolf, and dragons have to do with web design? More than you would think. Rebecca Mushtare discusses how a trip through fairy tales may open up the opportunity to develop empathy skills and conversations about race, disability and identity.

Allison Rank joins us again this week, this time as a guest host.

Show Notes

Transcript

John: What do the three little pigs, the big bad wolf, and dragons have to do with web design? More than you would think. In this episode, we’ll explore how a trip through fairy tales opens up the opportunity to develop empathy skills in conversations about race, disability, and identity.

[Music]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer. Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[Music]

John: Allison Rank, a frequent guest on this podcast, joins us today as guest host. Our guest today is Rebecca Mushtare who, until this episode, had been the co-host of this podcast.

Allison: Nobody panic. She’ll be back in this chair next week.

John: Today our teas are:

Allison: English breakfast tea.

Rebecca: What?!?

Allison: …under duress. I’m highly under caffeinated.

Rebecca: I’m drinking my normal English afternoon tea.

John: …and I’m drinking pineapple ginger green tea.
We invited you here today… because you’re always here… but we’re asking you…

Rebecca: …it’s a matter of convenience….

[LAUGHTER]

John: A year ago your daughter was born… now the three little pigs have invaded your class. Could you tell us a little bit about how the three little pigs made their way into your web design class?

Rebecca: I’ve been looking for ways to help students develop more empathy for their audiences, and it’s been a struggle. Students (or anybody who’s new to anything) will immediately try to make things for themselves, because it’s the audience they know best. So, it’s the easiest way. If you’re working on technical things or other concepts you don’t have to worry about audience too, because you have that part figured out. But, I’ve been really wanting to challenge students to dive into audience and also deal with accessibility issues which doesn’t come intuitively to them. So, the three little pigs actually offers a really great opportunity to have different audience members to think about (and audience members that don’t really exist); it becomes a safe zone. In this scenario, I’m using three titles as ethnographies for the students to read to get to know their audience better. I spent some time reading about ten different versions of the “three little pigs” and I’ve identified the best three. They are: The True Story of the Three Little Pigs by John Scieszka and Lane Smith, and The Three Pigs by David Wiesner, and There’s a Wolf at the Door: Five Classic Tales Retold by Z.B. Alley and R.W. Alley.
They read those books and then we come into class and I ask them to help me understand who all the characters are, what’s important to them, and some of their characteristics or qualities that we need to think about in terms of design… and then (from the perspective of the characters) what’s going on in the community that they live in… and the frame that I’m giving my students is that they’re in this community called Dragon Town. Dragon Town has a mayor named Mayor Melanie McDonald, and she’s human, but there are talking animals and dragons and other creatures that live in this community together and there’s a clear creature divide going on. So, the humans seem to value themselves more than the other critters in town. The poor pigs, they’ve got houses that are falling down. They don’t even up stand the Wolf’s breath. So, we’ve got some issues going on here.
The students read the stories, came to class, brainstormed about these characters, and helped identify some really big issues that were happening in Dragon Town… and then my challenge to them was, in teams of three or four, to identify one of those 10 that we identified as a class…choose one that they were gonna use a web design to help raise awareness of or to start to tackle. Obviously they’re not gonna solve these big problems, but they could make a dent into it.

John: The purpose then is to have students look at a problem from another perspective, from the perspective of the intended audience of the webpage, rather than using their own biases.

Rebecca: Yeah, exactly, and it’s something that they really need to practice… and so, yeah, this is a good opportunity to do that. They’re characters that their familiar with, but the books actually challenged a lot of their initial remembrances of some of the stories. So, it’s a nice way to get them to revisit that in a different way.

Allison: How was this different than how you’ve tried to approach the same topic in earlier iterations of the class?

Rebecca: In a previous episode, I think I talked about my simulated client project where I had these big company scenarios with the audience members being Oswego (the community that we live in) and they worked okay… but the students had trouble aligning themselves with older adults or middle-aged individuals who they just don’t seem to find relevant to themselves and even though these are individuals that are readily available in our community that you could interview and get to know, it was a struggle. We did a project in the fall, “The Voices of Oswego Veterans” project that we had a guest (Stephanie Pritchard) on who talked about that project… and we did a web project with that as well… and that was another way to deal with the audience. This time the audience was members of the Oswego community (the SUNY Oswego community), so they had a little bit easier access to that community… but the community that they were representing was different from themselves. These were students, so the population that they were addressing or talking about was student veterans, which was an identity that nobody in the class happened to identify with. That got us closest to solving the problem… but it wasn’t quite where I wanted them to be yet. What’s nice about this is that you don’t have to worry about offending anybody, because they’re not real.

Allison: I can imagine how the fictional characters are really helpful in terms of giving students a lot of space to play and a lot of leverage, but I have to imagine that there are some real challenges associated with giving them that amount of space as well. I guess I sort of have a gut reaction that thinks that they will make up things that cause problems in and of themselves. They’ve got enough rope to get in some dangerous positions. What are some of the challenges that you faced?

Rebecca: That’s a really good question. What I found was, they were willing to talk about things that they were never willing to talk about before. That, first of all, was a good space to be in. That was things like: “oh, there’s species profiling going on…,” “oh, there’s accessibility issues because pigs have hooves so they can’t type and tap on the computer screen…” …the accessibility issues that just bubble up. There was also the concern that critters were eating other neighbors, so we needed to start a campaign to be vegetarian, for example. So, there’s a lot of different things that came up…. a lot of social issues… another one was stranger danger… and then they did these presentations to the mayor, and it was important because we brought someone from outside in and I think that helped prevent some of the issues that you were identifying could bubble up as being a problem, but there was someone that wasn’t me who was the audience but I didn’t tell him who it was gonna be (it was just a grad student I bribed) who came in and just sat and played the part and asked questions and what have you…. and they were taking notes and then we went away and had a meeting and I came back with notes to the students about what the client was concerned about. So, that helped resolve some issues. But, you know, in the presentations there were some crazy things that happened… like the one on stranger danger, for example, the students had still indicated that the stranger, the bad character, was the wolf and the whole point was that all of the animals, and all of the creatures, and all of the humans, also have children and they all need to be concerned about strangers. That we shouldn’t associate one population as the bad actor. We ended up having to have a conversation about that. You can’t perpetuate these stereotypes, but what happened was we could have that conversation safely.

Allison: The familiarity played in the same way that a stereotype would traditionally function in class, but in a much safer space to have the conversation that resolves it.

Rebecca: Exactly. We were having crazy conversations about racial bias, and all these sorts of things, but under this guise of “it’s about the species” and the species problem that’s going on. And now all of a sudden it became safe. When that one group was having issues getting their head around it, I said to them: “You realize that this is the exact same thing as racial bias, right?” and they just looked at me with deer-in-the-headlights look. The next time they came back, the whole project was fixed.

Allison: That was actually gonna be my next question. At what point did you pull out from playing in the sort of allegorical space to say: “Hey, here’s what we actually just did” or did you let the experience and the skill building stand on its own?

Rebecca: I let things unfold organically, and I prodded and probed as necessary. I didn’t want any projects to perpetuate stereotypes or to perpetuate lack of accessibility… those two key issues. I probed and invaded their team time a lot with those particular things to push them on that, but you know they’re not perfect. But, I think they did a lot more growing in that area than they would have otherwise. What I think is missing, that I want to do next time is allow for more of that reflection at the end, so that they could apply it to some other projects. What I’m thinking about doing is have them present the work as if they were in an interview, and so how would you explain this project and what you learned from this project to a potential employer who has no idea what Dragon Town is, so that it becomes something that’s valid and useful… and I think that’s going to take some effort on their part to make that leap. But I think it’s actually a really good project for them to talk about in an interview and most employers would see the value in that.
I already have them do portfolio documentation. I already have them thinking about that, but I need to coach them through that process a little bit more…. and maybe actually make them present that.

John: Yeah, I could see an employer looking at a webpage making a case on avoiding inter-species consumption and being perhaps a little bit puzzled….

Rebecca: The tagline was “don’t eat your neighbor.”

John: Yes.

Rebecca:… which I thought was right on.

Allison: Yeah.

Rebecca: Well, and that group actually was interesting too because they wanted to do something that was: “Don’t eat your neighbor.” They wanted to be vegetarian but I was like, “Well, dragons have a big appetite. What are you gonna do for them?” So they came up with this tree salad or whatever that has just bigger things. They had to adapt the recipes and things like don’t forget there’s small kids. You got to think about these different populations, and they adjusted their content accordingly, to rise to that occasion. I also found this really great article about whether or not pigs are colorblind that I used as a doorway into thinking about accessibility issues. Apparently, I learned, pigs don’t perceive color the same way that humans do. They can’t perceive as many colors, so we have to really be concerned about the spectrum of colors and the kind of contrast that colors have… so that they would be accessible to pigs,,, but that led into conversations about maybe the pigs have to use voice activation because their hooves won’t let them type on their devices… and then we also had to talk about a mobile device for a dragon is pretty large…. so we had certainly some fun playful conversations, but they were really meaningful. We started talking about those issues pretty deeply in a way that I’ve never had in my class before.

John: Were the students more open to addressing these issues when it was in this safe zone or this safe space?

Rebecca: Yeah, even when I called that one group out on being stereotypical and perpetuating bias, they just received… and were like: “Oh, okay” and then you try it again… “is this better?” “My god, could you push it a little bit more?” and gave them some ideas about how they could push it… and our first solution wasn’t great after that…. It was to put in a separate monster that didn’t exist in this world as being the stranger, and then I identified that like when someone the other, we shouldn’t just assume that they’re the bad person or the bad creature. We had to be careful. I tried to call them out on whether or not we were using the word person, because it didn’t apply to dragons. So, it was funny [in] their presentations they were really conscious about things like that and trying to be inclusive in their language. So, yeah we ended up trying to tackle some of those things, and I was pretty impressed with how far they got… but it took some pushing. That one group took four or five tries before they had something that was gonna work.

John: How did students respond when you first gave them the assignment?

Rebecca: Well, I should probably provide a little setup in that my class includes design students, marketing students, and graduate students in HCI. So, it’s a fairly diverse population in and of itself in terms of disciplinary background. So there’s that. There are a number of people in the class who may not be traditionally artsy or creative, so it’s a little risky, right? I think I’m also known for being very serious. Which if you know me personally, that might not be true, but in the classroom students perceive me as being very serious… and the semester just was not going great, to be honest. It’s like something’s got to give, the students were struggling with a lot of the technical things, and so I basically threw the syllabus out or revised it significantly. stopped and did just technical exercises so students get comfortable with some of the things that they were really struggling with… and then one day I just showed up and said this is what we’re doing… and they had a ton of fun…. and were shocked… they’re just like “Is she serious? She lost it?” There was definitely those looks, but then there was a couple of key students who just jumped in and ran with it… and I think that really helped. So, I’m hoping that that will happen again. I think if everyone in the class is a little too serious, I don’t know that it would work.

Allison: Would you plan on sticking with, in the future, the three little pigs as sort of the through line story or it sounds like the story with the five different ways that the wolf is at your door? Does that give you some entree into some other storytelling avenues?

Rebecca: There is some entree into some other avenues and I maybe need to read some more fairy tales to be up on that, but the reason why I stuck with the Three Little Pigs is actually the wolf is the character that carries through all of them. So, that the five stories that are connected are all based on the wolf and different stories. So there’s Little Red Riding Hood, the Boy who Called Wolf, those are some of the stories in that other one. So, maybe there’d be some versions? I also happen to know that there was like the version of the Three Little Pigs told from the wolf’s point of view, so I really like that because it’s in direct conflict with the Three Little Pigs version of the story. I liked that the ethnographies that they were collecting were realistic in that they conflicted with one another, that they had to deal with the fact that there was conflicting information, and that they had to resolve that or deal with the fact that a wolf’s perspective was different than the pigs perspective of what the wolves perspective was… and I think that was a healthy messiness about it that worked pretty well… and the particular version of the Three Little Pigs that I used pigs escaped getting eaten by the wolf because they jump out of the storybook. So, there’s some plot twists in there that the students wouldn’t necessarily expect. It’s not a traditional version of the story… plus, they all have really great illustrations and they’re beautifully designed.

Allison: Are there other classes where you’d be interested in trying the same type of fictional ethnography technique?

Rebecca: I think it could work in some other scenarios, but I like this because it’s in my intro class. It’s a nice doorway in. What I’m really interested in seeing is, when I have a couple of these students in the advanced class next time, if that impacts their ability to do some actual real audience research and use that research in context. I think I want to monitor that first before doing some of this other work. I like it in particular because it’s a beginning class even though it’s at the 300 level.

John: It sounds like a really fun project, and there’s nothing really wrong with making learning fun.

Rebecca: Yeah, I had a good time and we had some moments where you had to really practice the deadpan look, you know, be really serious about what it is that we’re doing… and that part was really fun.

Allison: …and that seems like an amazing turnaround on a class where you have to scrap the syllabus halfway through a semester.

Rebecca: Yeah, it was amazing… the community that was formed around the project… and the way that they were exchanging with one another and coming together was incredible, and I was so thankful.

[LAUGHTER]
There’s nothing worse than an off semester and you just want out. I think everybody wanted out and so I just said “We’re out. We’re gonna try something new” and it worked, so that was good.

John: I guess the next question is: “what are you going to do next?”

Rebecca: That’s a good question… I think that with this project I’m hoping to expand it a little bit… so I’m currently thinking through “are there things that I can eliminate that I was doing before that I could embed in this project or I just allow them to have the time and space to fully build things out?” They have really good ideas and pretty good plans and the execution is almost there and I’d like to be able to have them have that time for the “almost there” to be “there” and then also to do that reflection piece that I kind of half-assed.

John: Okay, well thank you for joining us and I guess we’ll see you again on our next episode… and back as a host.

Rebecca: I mean, that is, if you’ll have me back.

[LAUGHTER]

[Music]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[Music]