Category: clickers

422. Negative Examples

Posted in active learning, clickers, edtech, educational technology, evidence-based practices, evidence-based teaching, higher education, learning, pedagogy, teaching
December 3, 2025

To understand what a concept is, we must also understand what it is not. In this episode, Bill Goffe joins us to discuss his study of the effect of the use of negative examples on student learning outcomes.

Bill is a Teaching Professor in the Economics Department at Penn State.  He had previously been one of our colleagues here at SUNY Oswego. Bill is very well known in the profession for his Resources for Economists on the Internet, which was one of the very first internet guides available for economists, and it’s now hosted and sponsored by the American Economic Association. He has served as a member of the American Economic Association’s Committee on Economic Education, the Secretary-Treasurer for the Society of Computational Economics, an Associate Editor for Computational Economics and the online section of the Journal of Economic Education. Bill is currently co-chair of the Liberal Arts Teaching Group, a faculty learning community at Penn State.

Show Notes

Transcript

John: To understand what a concept is, we must also understand what it is not. In this episode, we discuss a study of the effect of the use of negative examples on student learning outcomes.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Bill Goffe. Bill is a Teaching Professor in the Economics Department at Penn State. He had previously been one of our colleagues here at SUNY Oswego. Bill is very well known in the profession for his Resources for Economists on the Internet, which was one of the very first internet guides available for economists, and it’s now hosted and sponsored by the American Economic Association. He has served as a member of the American Economic Association’s Committee on Economic Education, the Secretary-Treasurer for the Society of Computational Economics, an Associate Editor for Computational Economics and the online section of the Journal of Economic Education. Bill is currently co-chair of the Liberal Arts Teaching Group, a faculty learning community at Penn State. You can also find Bill on many listservs and forums devoted to teaching and learning. Welcome back, Bill.

Bill: Thanks, John. Appreciate that.

Rebecca: Today’s teas are:… Bill, are you drinking any tea with us today?

Bill: I am. I’m drinking Twinings green tea, decaf.

Rebecca: Nice, good choice. How about you, John?

John: I am drinking a peppermint tea from the same vendor.

Rebecca: Nice. I have hot cinnamon spice today because my daughter requested a pot of it. So we’re sharing a pot of hot cinnamon spice this afternoon.

John: Very nice as we approach the holiday season here when we’re recording. So we’ve invited you here today to discuss your recent working paper with Austin Boyle entitled “Negative Examples in Lecture Improve Student Learning. First, can you tell us what you mean by negative examples?

Bill: Sure. So you’re describing a concept to someone, let’s say a young child, and describing what a dog is, and you talk about different types of dogs: Dalmatians, Pomeranians, whatever. And unless you describe how a cat is not a dog, they’re probably not going to have a great idea of what a dog actually is. And this might be the case, say, for a second language learner, where you might have different terms for different types of mammals, also related to something called contrasting cases, where experts are very good at delineating different types of phenomena or different types of facts, if you will, say a physician or variation theory used in math. So these are all ways of getting students a rich understanding of something.

Rebecca: Can you talk a little bit about what prompted your interest in negative examples?

Bill: Sure. So I’ve used clickers to teach for a long, long time. You currently use iClicker Cloud. Love it. And I tried to write a paper on just in general, using clickers to teach economics, and the paper didn’t really work very well. There’s reasons it’s never been written, and the reason was I use clickers in many different ways, so a little bit more focused approach to having a paper. And a few years ago, I came across the book How to Teach Economics by Yousuf Hamid. He was a high school teacher in Great Britain, and I had to read that. Well, to be honest, I read it during graduation. You have to go to graduation every year, but, you know, it always kind of work. And he talked a lot about negative examples. And I thought, “Oh, that’s pretty interesting.” And it also followed work by Kirschner, Sweller, and Clark on the idea of direct instruction lowercase “d” and “i”, and I thought, “Well, that’d be an interesting to try, and it’s fairly focused,” and that’s the genesis of the paper, and pulled Austin into it as well.

John: We’ll include citations to those resources in the show notes. So, many people, when they’re teaching, tend to use primarily positive examples. What’s the problem with just using positive examples of a concept? I think you’ve already addressed that a little bit.

Bill: One example, the original work on this, at least formal original work… the idea goes back a long, long time… it was by Engelmann and Carnine, a book on direct instruction. And they give an example of a glem, a G- L-E,-M. Could you define that just with one example? Well, you could not. And they give an example of a pencil. Well, is it wood, or is the pencil horizontal, whatever? But it helps you focus in better on an idea when you understand what it’s not. For example, in economics, we talk about capital quite a bit. Unless you tell students, it’s not stock, they probably won’t pick up on that. Or you give examples of money, how money is very specific to economics. To economists, they’re probably going to pick up on it’s not some general sense of wealth.

John: And I can affirm that the concept of capital is one tht students have always had a lot of trouble about because the way we use it in economics differs from how it’s used in everyday use. And just for our listeners, could you explain the difference between the everyday use of capital and the way we use it in economics?

Rebecca: I’m an everyday listener.

Bill: Great. We’ll see if this works. So, to an economist, capital is something owned by business, manufactured and used to make things, so, a commercial mowing machine, a commercial oven, a jet liner, construction equipment. In everyday usage, it might also include financial capital, stocks, for example, might be considered capital. I would add one thing economists really haven’t done, the Bureau of Economic Analysis actually redefined capital recently, includes intellectual capital, and I just don’t teach that. It would be confusing, but there’s actually about as much intellectual capital these days as physical capital in the United States. But maybe in an intermediate course. I would do that.

Rebecca: Seems like an easy thing to get confused. Can you talk a little bit about how you pick which concepts to use negative examples for? I can imagine that some concepts are more ripe for this strategy than others.

Bill: Yes, we wanted something fairly limited, not a long concept of how to manipulate a graph or something, that could be a further work, but just a definitional sort of thing, but the students have misconceptions about this, as we described there with capital. And so we picked three different ones, money, capital, and technology. Technology in everyday use, our phones, for economists, it’s basically efficiency in the production process. You know, it is rather unfortunate, there’s what, 50,000 words in the English language, and we use words that are used in other ways just to make it more complicated. And I joke to students, I wish I could use different words, but that would hinder their further education, I fear..

John: And this has been a problem for economists for well over a century in those very same issues, and it would be nice if we didn’t have to do that, because it introduces that confusion right when students are just starting to learn the vocabulary of economics, and the way I talk to them about I say, “Hey, part of this is you’re just learning the language of economics, which may look similar to how we use these words in everyday use, but it’s a bit different, and you need to focus on these things because of the ways in which we’re using the term.”

Bill: I would add that I’ve seen some STEM education research papers. They teach concepts first and jargon second, and they find that actually improves learning because you’re trying to remember the terms and they’re using the terms, it overloads your working memory. And there you teach jargon later, they find improvements.

Rebecca: We do that sometimes in design, where we separate the technical skill from the concept, because learning them all at the same time can be complicated. I was thinking of a term that I run into all the time where there’s confusion, where this strategy might work well, which is digital accessibility, where people confuse the idea of meeting these standards to help people with disabilities gain access to content and material with the idea of “just available,” like it’s on the internet.

Bill: I learned that distinction just recently myself. I would add on choosing these terms, we found even after instruction, students got it right between 60 and 71% of the time. So we seem to have picked a nice set of terms and definitions to use.

John: Which is actually much better than they would tend to if you had only used positive examples. And we’ll talk about those results a bit in just a minute.

Rebecca: Can you tell us a little bit about your study? How you had it set up? How many students were involved?

Bill: Yes, we had about 1200 students. I teach large sections, about three sections every semester, about 340 students that did this over two semesters, basically just repeated things, get more power for our tests, and did a so-called crossover study. Crossover study, have your participants that are both controls and the intervention side of things. We did a 3×3 one, which is a little bit hard to describe on a podcast, where you can’t visualize things, but there’s a fairly famous one for ultra processed diets. You might have heard of by Hall and company in 2019. They put people into hospitals, they control directly what they ate, and one group got ultra processed foods for two weeks, the other group got more wholesome foods, I guess you could say, and then they switched who got the wholesome foods, who got the ultra processed foods. So thus you could deal with the fact that people might have responded differently, that different foods have different effects on people. And we did the same thing here in our study, except we had three concepts and three sections and three teaching methods.

John: So what were the three teaching methods that you used in the study?

Bill: So we just use a positive example, where I would list a half dozen examples of capital or money, or not quite of money, because there are quite six examples of that. Then with lecture, we had both positive and negative examples, where these things are money, this is not. This is capital. This is not. And a third type of teaching method was with a cliicker question, where I gave a student’s list of things. Some are positive, some are negative examples, and they just choosed which was which.

Rebecca: So what did you find? [LAUGHTER]

Bill: We found that including negative examples improved learning in the following class by about 21 percentage points. Let’s say the class got 50% correct just with positive examples, the class that got positive and negative examples got a 71% correct. So it is very statistically significant, it was robust with lot different statistical methods we used. So we’re pretty pleased with that, it was a clean result. Sample size helps, and I think probably doing it just with precise definitions, not long drawn out concepts or something like that.

John: Based on these results, would you suggest that perhaps we should all focus a bit more on using negative examples as well as positive examples in our teaching?

Bill: Yes, most certainly it’s an easy intervention. For example, you might know Jim Lang’s book Small Teaching, where he talks about easy interventions you can do, and this one’s easy, as long as you remember to do it. But you’re making up your notes, you think of things that aren’t whatever you’re talking about. You’ve probably seen those on exams, overall. Interesting result we had too that the clicker questions, the active learning, was no different than the lecture. So we’d like to do further work on that at some point, most active learning studies have been semester long, active learning not very well defined, and we didn’t find any difference here, so presumably, at some point, active learning make a difference be interesting to find out what that difference might actually be.

John: That’s an interesting result, and that’s probably worth some follow up.

Bill: I think partly on if you do see similar results with the flipped class and active learning and such, people don’t really have a framework on what they actually ought to be doing before class and what active learning should look like. I’ve got another paper on this, I mentioned, that uses so-called deliberate practice, That’s the 10,000 hour rule. It’s really not 10,000 hours, but there’s some key parts of active learning people don’t do that much. One is to focus on sub skills, for us in economics, things like change in demand versus change in quantity demanded. We generally don’t teach that specifically, is my thinking. And then also, you want to make connections between things. How are supply curves and demand curves separate? How are they similar? Because we remember things in terms of frameworks or schemas. The negative examples help develop schemas. You understand what things are not, kind of triangulating, if you will. And good active learning does the same thing, but if they’re just filling in a worksheet with numbers, they plug into the formula, that’s active learning, I suppose, but not much learning there. So we ought to focus more on exactly what active learning is doing for people.

Rebecca: Following your study, what are some different ways that you’ve incorporated this practice into your own teaching practice?

Bill: So I definitely try to do the negative examples, capturing, oftentimes what I see on these exams and when I’m thinking of a question, I always try to have it connected to something else. Oftentimes one concept’s related to another one, and we make that connection explicitly in clicker questions so that, “Oh, if you remember this, this is a variation of that.” For example, how is consumer price index similar and different than the GDP deflator would be one thing we do, and why are they fairly similar? To understand better how those things work. Oftentimes, I explain things based upon what they’ve done before. Another thing I do is we have very nice technology at Penn State, I have a main screen in the classroom. I have a side screen. Side screens, I put up an outline includes both the outline for the day and our bigger topics. So I’m constantly referring to that outline saying “what we’re doing here is a variation of this, is different than that,” and so forth. And if I do surveys as students, they find that very useful. A better study would be to not have the outline there someday and see how learning goes. I just now thought of that as an idea to do a study on.

John: We know that students come in with a weaker set of connections than faculty, and sometimes we assume that they’re able to make these connections and see the relationships, but they haven’t yet developed that skill, and sometimes it’s generally believed that that type of scaffolding, by putting up an outline of the main concepts you’re trying to get across, can help. But I don’t know if I’ve ever seen any tests of that, certainly not in economics, but I’m not sure if I’ve seen it in other areas either.

Bill: I have not. I can’t say I’ve looked very hard. I would add, to do my literature searches. Now I oftentimes start with ChatGPT, but that’s another whole podcast, [LAUGHTER] I suspect.

John: Yes, and there’s some other good tools out there, too. There’s research rabbit and other things that will actually, even in some cases, will provide more accurate results. But ChatGPT is getting a lot better all the time for these things.

Rebecca: Other than in the test results, have students responded? Or have you asked students about using the negative questions and how that impacts their learning?

Bill: I have not actually, that’s a good point. That’d be another thing to add. It’d be interesting see if they actually can pick up on that. Oftentimes, they’re not that great on understanding their own learning. Something we’ve all seen as instructors, someone thought they did much better on the exam than they actually did, which is something you want to avoid. That’s a tough place to be in and you have to be sympathetic as instructor when that occurs.

John: Now, one thing to keep Rebecca happy, we should also ask a little bit about the methodology that you used. In the study, you used both pooled data as well as panel data, and you also used ordinary least squares and binary dependent variable models. So could you just talk a little bit about why you used multiple approaches and how consistent were the results across those.

Bill: Yeah, the results are very consistent. So sometimes you don’t quite know what econometric approach you take, you can argue this is better, that’s better. And another approach, and it’s pretty common these days, I think it’s by far the most common, is you take multiple approaches and see if the results are robust, to see if you get similar results. Indeed, we did. So it’s a very happy occurrence when that occurs.

John: So were you surprised at the magnitude and strength of these results? I know when I read the paper, I was surprised at just how large that effect was, because it’s very rare to see effect sizes that large in a study of alternative teaching approaches.

Bill: I don’t think I had many preconceptions here. That’s odd. I certainly was pleased, but I can’t say I was quite surprised. It certainly makes me want to take this approach again with where we do the crossover studies, you can do many things with that. For example, we’re told we should teach about things students find interesting. Well, is that really true? I did an example of that this semester where Penn State students, they do a large fundraising project called Dance Marathon, or THON, and last year it raised $17 million, about half the student body is involved. And so I do a question about real prices, and the question was, did THON actually raise more money when you correct for inflation, you know, something they’re invested in a lot, and informally looking at the data, that didn’t seem to have any impact on that segment, on calculating real prices, I need to do more work on that, but I hope more people take this methodological approach, the crossover studies, where you could study a lot of things, the outlines, is it interesting or not, and so forth?

John: Because with the crossover study, you’re ruling out the possibility that there was just some systematic difference between the sections or between the other conditions, right?

Bill: Yes, exactly. We actually found there were differences between the sections. Some did much better on the questions than others, but by effectively pooling these different interventions and controls together, we can adjust for that. We can correct for that. I don’t see it that much in economics. John, have you seen it very much, the crossover studies?

John: I haven’t seen very many, but one that stands out, and I think it was done by economists, was back when the initial studies came out that suggested that when people took notes by hand, they remembered more than when they used a computer. I do remember a study, and I’ll include a citation to this in the show notes. What they did is they had all the students use mobile devices for note taking for a portion of the semester, they broke up the group into two halves, and half of them, for the first half of the semester, took all the notes using mobile devices, and the other half took notes by hand. And in the second half of the semester, they switched, and what they found was there was no difference between how students did when they took notes by hand and how those same students did when they took notes on the computer. However, those students who regularly chose to take notes on a computer, on average, did less well regardless of whether they were taking notes by hand or on a computer, which suggests that the students who choose to take notes by hand may be perhaps a little bit more focused on the material no matter how they’re taking the notes, and those students who may bring in mobile devices to use, perhaps may do it because they prefer having the availability of distractions. So it’s not quite the same thing, but it’s similar in that you’re controlling for the individual effects and separating it out so that you are getting a better comparison, correcting for any self selection in the people who are involved, which is really, I think, what crossover studies are trying to remove, the effect of any systematic selection between the groups.

Bill: I would add, I’ve surveyed my students, and generally my students aren’t performing quite as wel do choose to use laptops, and higher performing students do not, so consistent with your more serious study, for sure.

John: And another thing I’ve been noticing a lot is that for quite a while, I’ve been walking around quite a bit, but I’ve also noted that the students in the front third or so of the class tend to be very actively engaged in answering questions, and they tend to do really well, and they tend to ask more questions. But when I’m in the back of the room, I see a lot of laptops where there perhaps may be a video playing, there may be some online shopping going on, or there may be some sort of video game that they’re working on on the mobile device and their laptop or one or the other. And I think we often do see that sort of sorting in terms of the way in which students choose to work. I’m not sure how I could get rid of that effect, other than moving them all up to the front.

Rebecca: it does seem like at least the multi taskers, in quotes, are at least removing themselves, and because they’re probably distracting to other people, they’re just moving themselves to the back so they’re not a distraction to the people who are trying to focus.

John: Except they’re surrounded by other people, and there may be a bit of a contagion effect going on back there.

Bill: I’ve seen papers on the contagion effect, yes. There was a Carl Wieman paper. They did look at seating in large classrooms and found a very definite effect. They even switched seats halfway through the semester to bring the people in the back up to the front, and they still did worse.

John: Some of our students are just much more interested in the material and much more engaged than others, and there’s just so much we can do with that.

Bill: It’s tough, but I would be a little bit sympathetic to students. It’s been a long time since I’ve had to sit through a typical undergraduate today, and you know, you’re three, maybe four courses a day, a lot of sitting. That would be tough, a challenge. There’s a book called My Freshman Year, and there’s a field anthropologist who went to college for a year to understand her students better, and she found it was harder than she thought.

John: And she found some dramatic differences, too, in the way in which students connected with each other from when she had been a student, that they had much tighter networks with the people they knew before they came to college, because it’s less costly to connect with them over social media and so forth.

Bill: Yeah, friendships have changed with the rise of social media.

John: It used to be costly. I remember having to bring all these quarters to put in pay phones back when I was in school.

Rebecca: You mean, in those 1900s phones that my daughter refers to, John?

John: Yes, I think that would be the case.

Bill: What’s a quarter?

Rebecca: Yeah. [LAUGHTER] Well, I think we still know what quarters are. We just don’t know what pennies are anymore. [LAUGHTER]

Bill: Well, that could be, yes. [LAUGHTER]

John: Are there any other things you’d like to share about this study?

Bill: Well, I think that’s pretty much it. I think I would emphasize this little bit more about, again, developing student schemas. But we talked about that earlier today, this broad network, and it’s through connections, and this helps them make connections more focused on whatever the topic might be at hand. So I think negative examples could be used for teaching most anything.

Rebecca: Including our colleagues when we’re doing professional development and all kinds of other contexts.

Bill: Yes. [LAUGHTER]

John: And it’s easy for us to lose focus, because we already can distinguish these things more easily, and students have not yet developed that capability.

Bill: The curse of knowledge really reigns here, where you know it well. It’s hard to understand how other people don’t. I was doing a think aloud exercise the end of last semester where I interviewed students half the semester in how they’d interpret different things. And I spent a lot of time on aggregate supply, aggregate demand model in my course, standard staple things course, asked about certain phenomena, no one actually used the model. I was only interviewing four students, so, I mean, it’s not much power there, but it was kind of crushing, but it gave me insights on how they think about things.

John: In an earlier podcast with Doug McKee, he talked about having students come in and just work through the problems verbally, so he could see what sort of process they were using, and by seeing that, it was easier to help guide them in a more productive direction. Because if we don’t really understand how students are processing things, it’s hard to help them understand things more deeply.

Rebecca: Yeah, we use that talk-aloud method and design a lot too around troubleshooting, because troubleshooting strategies require, especially around technical skills, you need to think about problems in particular kinds of ways to be able to solve them correctly, or to be able to solve them at all, [LAUGHTER] and doing that kind of talk aloud strategy works really well for figuring out where something went awry in the troubleshooting.

Bill: And experts think a lot differently than students, I would suspect. You would never think someone thought that way.

John: And fMRI studies confirm that different areas of the brain are activated for experts and for novices when trying to solve a problem. So it’s easier when you become an expert, but we have to help students develop the skills to get there. While we’ve got you here, could you also tell us a little bit about how you teach classes so large with so many students, every semester.

Bill: I use Perusall as a way to have them read ahead. They come into class, they have some sense of things and then a lot of clicker questions, typically, as long as it’s not something new. For example, I was doing long-run growth. I do that a fair amount, and the book doesn’t really talk much about the institutions and the role of markets in that. So that was a day or two of me talking. I’m going to be doing the business cycle starting tomorrow, and that’s a fair amount of me talking. But other than that, it’s them pretty much asking questions, or me asking questions. Probably on Thursday, we’ll get to calculate employment rates like read about that in Perusall. I’ll ask them, “Who in this group of people is unemployed?” Then I’ll show them some data, what we compute from that data. So it’s a lot of using ideas. I do a lot of peer instruction where they first do votes on their own, and then I say, “Okay, there’s a lot of uncertainty here. Please convince your neighbors.” And I use an app by iClicker, so I can see their votes on my phone, and then they do a revote, and they usually improve 20, maybe 30 points, then I provide further instruction on the topic at hand. And there’s a couple papers I’ve seen. Let’s see Michelle Smith at Cornell, a computer science paper. They found that if you have students revote, that helps them; if you have a further explanation from the instructor that helps them; if you both, it helps them even more, and it’s probably enriching their schemas, they’re seeing several explanations that reinforce the connections between things. Also I do a series of quizzes, rather than midterms. I do seven quizzes. We’re very fortunate at Penn State, we have a testing center, so I send all my students there for testing. So I run the quizzes for two days, after the quiz they do a second stage of the quiz. In class they start, then they answer at home using any resource, and finally I release the answers, and then they reflect on and describe what they found most difficult on that quiz. So we do a two-week cycle. There’s a lot of feedback, even though the course is quite large, with about 1000 students a semester.

John: So having that testing center gives you a bit more security in the assessments than many of us have when we give quizzes online. Many of the assessments I use are done online, and I used to feel a whole lot better about them before we had ChatGPT and other AI tools appear, especially with some of the browser extensions, which will directly answer things, so you’re able to avoid some of that, which is really nice.

Bill: I really feel very fortunate, yes. I would add, it’s a good thing that the AI tools came after Covid, rather than before.

Rebecca: So one of the things that you mentioned when you’re describing your class was Perusall. Can you describe what that is for people that don’t know?

Bill: Oh, yes. An e-text goes inside perusal, and then students mark it up with comments, questions, and responses to other people’s questions. So it has them thinking bit about the text. It grades them automatically. They need to start early so there’s a little bit of spacing there. They need to make sure they document across the assignment, and they’re also rewarded for opening up multiple times, not just the evening before the exam. And it’s fairly straightforward. Most students get full credit on each assignment. So it’s a bit of encouragement there to read ahead. I used to use so called JITs for just-in-time teaching assignments where I’d ask some questions, including what they found confusing, but I like Perusall better. It’s less grading work, and also, I think it covers a little bit more topics or not just looking at the answers to my JIT questions.

John: Now, do you break them up into groups when they’re using Perusall?

Bill: Yes, Perusall actually does that. When I look at the marked up text from 1000 students, the page does not load basically, or has difficulty loading. And you can also download, I’ve not done this, you could download all their comments, and you can run through a chatbot or whatever to get a sense of what they have trouble with, or what they find confusing. I think textbook authors would find that really, really useful. You could say your textbook was designed with the input from 1000s of students, and so your students can read this more carefully. I’ve not found an author willing to do that, but I think that would be a big win for someone.

John: It could be, and from a marketing standpoint, that could be really helpful. When iClicker announced that they had AI tools being built in, I was really hoping that they would integrate that into the text responses, because right now, it’s just generating questions, which is not bad, and it doesn’t do a really bad job of that, but it would be so nice when you got maybe 1000 or more responses coming in, or a few 1000 responses, if it could very quickly generate a summary of the responses to free-response questions. That seems like something that wouldn’t be all that hard to do, and it would be a nice extension.

Bill: I agree. I think there’s some polling technologies that do that. For some reason iClicker doesn’t. I wonder if there’s a legal issue or some other constraint. I would love to see that as well so I’m not just asking five different possibilities.

Rebecca: So we always wrap up by asking: what’s next?

Bill: So I want to do more work on active learning, the ideas from deliberate practice. Does it make a difference if students have good sub skills. And does it make a difference if you teach by making connections between different topics? There’s many ways active learning are implemented. And it would be cool to do a study where you look at active learning in many different implementations, but some get those interventions, some do not, in a crossover study, of course. So that’s what I would like to work on next a bit.

John: That sounds like a very productive area of research that could really be helpful for all disciplines.

Bill: Yes, well thank you. That’s the idea, I think, a bit about marketing of papers, you know, who would find it interesting, and so forth.

Rebecca: Well, thank you so much for joining us, and we look forward to your future research.

Bill: Thank you so much. It was great fun.

John: It’s great talking to you, Bill, always. I’ve learned a lot from you over the years, and it’s great to still be able to see your work out there in various places.

Bill: Great. Okay. Thanks so much, John. It’s great to connect up again.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Ryan Metzler.

[MUSIC]

347. CATs and AI

Posted in access, clickers, diversity, edtech, equity, feedback, first-generation, higher education, inclusion, learning, pedagogy, teaching
June 26, 2024

Classroom assessment techniques, initially developed at a time when chalk-and-talk instruction was the norm, helped to shift the focus from teacher-centered to learner-centered instruction. In this episode, Todd Zakrajsek joins us to discuss how generative AI can enhance these techniques by providing more immediate feedback.

Todd is an Associate Research Professor and Associate Director of a Faculty Development Fellowship at the University of North Carolina at Chapel Hill. He is also the director of four Lilly conferences on evidence-based teaching and learning. Todd is the author of many superb books, and has published five books (so far) in the past five years. His most recent book is a 3rd edition of Classroom Assessment Techniques: A Handbook for College Teachers, with Thomas A. Angelo.

Show Notes

Transcript

John: Classroom assessment techniques, initially developed at a time when chalk-and-talk instruction was the norm, helped to shift the focus from teacher-centered to learner-centered instruction. In this episode, we explore how generative AI can enhance these techniques by providing more immediate feedback.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Todd Zakrajsek. Todd is an Associate Research Professor and Associate Director of a Faculty Development Fellowship at the University of North Carolina at Chapel Hill. He is also the director of four Lilly conferences on evidence-based teaching and learning. Todd is the author of many superb books, and has published five books (so far) in the past five years. His most recent book is a 3rd edition of Classroom Assessment Techniques: A Handbook for College Teachers, with Thomas A. Angelo. Welcome back, Todd.

Todd: Thank you, Rebecca.

John: It’s nice talking to you again. It’s been a while. I think the last time was last summer.

Todd: Really? It seems like yesterday. Time goes by. It was last summer. You came to my office and recorded.

John: That’s right.

Todd: It was fun. You left me a really nice set of tea. Wooh.

John: And a mug.

Todd: Yes, I have a mug for the show. [LAUGHTER]

John: Speaking of teas, [LAUGHTER] today’s teas are:… Todd, are you drinking tea?

Todd: Every time I’ve been on this program I’ve been drinking tea and I’ve had some pretty exotic brands and blends but, I’m sorry, today I have water, the lightest of all teas.

Rebecca: It’s a very light tea. I have English Tea Time tea.

John: I have Lady Grey today.

Rebecca: That’s a good choice.

Todd: Alright, I think I came in third place.

John: We’ve invited you here today to discuss classroom assessment techniques. The first two editions to this book were written by Thomas de Angelo and K. Patricia Cross. In last week’s podcast, Tom joined us to discuss the origin of this book, where you joined as a contributor for the current third edition of this classic book. What was your reaction when you were first asked to join in this project?

Todd: Well, this is actually kind of a funny one, because it goes back to Linda Nilson and Teaching at its Best. And Linda was retiring, and the publisher, Jossey-Bass wanted to do a new edition of Teaching at its Best, and Linda says “Why don’t you see if you get Todd to do it and I will work with him on it and do it.” And we did it. And we got the book done a little bit early and on schedule, and it came out very, very well. So there’s a little plug for that book there. And it must have been about two months later, I had called them and said: “If you find another project like this, I love working with people like this.” And a couple of months later, I got a phone call and they said “We have another project we’d like to potentially have you work on” and I said: “Great, what is it?”… and they said: “Have you ever heard of the book Classroom Assessment Techniques?” And I held it together, and I said, “Well, yeah, yes, I have.” And they said, “Well, we thought it’d be helpful given what you do and what Tom does and everything, is maybe the two of you could work together on this book.” And so he talked a little bit about what there was and everything, and I hung up the phone and did a little happy dance because this book was one of the most influential books in my entire career. But the way the universe works is I ended up doing this one because of the work with Linda Nilson’s. And so that’s how I got started on this.

Rebecca: That’s amazing.

John: And, I agree, that is an excellent book as well. So you’ve got two classic books that you’ve worked on recently.

Todd: It’s really surreal, you know, a first-generation college student and every once awhile is kind of like I just can’t believe this is happening. But it was a really good project.

Rebecca: When did you first start using classroom assessment techniques in your classes? You mentioned that this book was influential.

Todd: I taught my very first course in 1987-1988. This book came out, second edition, 1993. And I was using it pretty much the moment it came out. Somebody had pointed it out to me and I didn’t know anything about classroom assessment techniques, and then I started reading through it and picking some out. I thought it was genius. Much like periodically, what you’ll do is you look at something and think this is ridiculous that nobody started doing this before. In some respects, it’s not that hard, right? You ask people questions, and you find out if they’re learning. It just took Tom and Pat to come along and say, “Why don’t we come up with strategies to find out if they’re actually learning.” So I think it was really influential for me because it kind of struck that chord with me that I always believed we should be finding out how much students were learning instead of focusing on teaching. But that’s not what was going on at the time, everybody was focused on teaching-centered educational practices.

John: What do you find most valuable about these classroom assessment techniques?

Todd: It’s interesting, because there’s a couple different levels. The one that’s really valuable, of course, is you find out if students are learning. Secondarily, you can find out if you’re teaching well, and you can change your teaching practices based on what they’re learning and based on what’s going on. But I’ll tell you the real reason I think this is really, really valuable is it addresses equity issues very, very cleanly. We walk into classrooms with all kinds of preconceived notions and stereotypes and implicit biases. And it’s really, really easy to think that a student’s struggling for a certain reason, unless you ask them. And so, for several of these strategies, I think it’s great because I’ve had students that have been practically sleeping through every class and then I started doing the classroom assessment techniques and he had some of the best responses, this one student did. And I would just totally blown away that this person was just processing it to such a level and I had a preconceived notion based on where the person sat and how they dressed and how they acted in class. I didn’t think they knew anything or they didn’t think they were learning and they really were. So I think it’s huge for equity and overcoming some of the biases we have in our classrooms.

Rebecca: That’s a really good example to underscore the importance of that. We had the opportunity to ask Tom what his favorite classroom assessment techniques are. So I’m really looking forward to asking you the same question: what are your favorites?

Todd: I’m one who says over and over again, in education, that we should always be careful about absolutes. Like, there’s never a strategy that always works. There’s never a type of student that’s always the best. Any situation can change. So there’s like 55 CATs in there. And I listened to the program with Tom and I heard him say that he didn’t have his favorites. And then he quickly picked out his favorites, and I’m sure it’s the same with his daughters. [LAUGHTER] So I have three girls, you know, they changed the favorites, I would definitely go with the standards of the muddiest point minute paper, those two are the big ones. Everybody’s done a minute paper, probably at some point. The muddiest point I like, which was the adaptation. And if you haven’t heard this show with Tom, make sure you listen to that, because he explains that very well. I think I like exam wrappers where you go in and kind of have the students talk about the exam experience. But for me, there’s several CATs that are based specifically on assessments. And it’s the thing that we almost never ask students for, and it’s the thing they can give us the best feedback on. So it’s like, when you have group activities, it’s an assessment on how well the students perceived that the activity worked for them. So instead of me just saying, we did a great group project today, you ask the students: “Was this a good group project?” There’s assessments in there about learning interests, what types of things you’re interested in, your study strategies, exam strategy. So you can ask students those things. And I think those make great classroom assessment techniques, to find out the processes that you’re doing, and how the students are receiving those.

Rebecca: Those are good opportunities to respond in the moment in your class, but also probably plan in the future too, to know what to do next time around, perhaps.

Todd: Yeah, I think specifically for exams. And one of the favorite questions I’ve always had on exams, because the students would get to a point where, and I got this, because of one of the assessment techniques I’d use early on, is students always were frustrated that there was something on the test that they studied really, really hard for, and it wasn’t on there. The problem was, it wasn’t always the same. So the students all had different things they studied. So because of collecting that information, and finding out from the students as they were frustrated, I started adding a new question at the end of every exam worth the same as some of the other questions there, was: “Please describe in detail something that you study that I didn’t ask you about on this test.” And students immediately felt better about the exams, they were happier with the test, because they didn’t spend four hours memorizing something that they perceived wasn’t valued on the test, even though the real reason for all of this stuff is life, not the test. But again, that was one from just asking the students: “What do you like and not like about the test?”

John: That’s a really great application of a technique that I hadn’t thought about, I can see how students would feel that their voices are being heard with that type of thing.

Rebecca: …or that their efforts were validated.

Todd: They got their points for their studying, and it just really seriously changed the affect of the class.

John: In the last podcast we did with you last summer, we talked a little bit about how teaching has changed from the time when you and I both started teaching, and this book was written back around the time when you and I were both relatively new at this. What role do you think this book has played in that evolution of instructional practices?

Todd: Well, first of all, I do want to acknowledge and help people just really understand the difference between 1993 and 2023. Some of the things we took out of the book was “display your results on an overhead projector,” there was no mention of the internet anywhere in that book, because the internet wasn’t here yet. Just stop and think about that for a second. So it was a very different time. A lot of heavy lectures. I mean, a lot of times people were lecturing, there were other activities, or other strategies, there was group work and role plays and those types of things. But primarily people are lecturing, because that’s just what people did, primarily. And so the role that this played, I would say, I don’t know if I’ve ever said this before. I think this book had a bigger influence on higher education than any other single book that’s out that I know of. There are some really, really good books out there. But this fundamentally changed the game. Because what it did was it started asking students about their learning, which means it started to really shift the focus from teaching to learning. This did happen to come along, by the way, if you really look at dates, the whole concept of really pushing active learning in higher education, at least in the United States, really took off in the mid- and late-1990s. So 1996 to 1998, we started seeing people like Richard Hake, do some huge studies about this. American Association of Higher Education was changing their themes from taking teaching seriously to taking learning seriously, all of those things happened about three to five years after this book came out. So I think this was the impetus for a huge, huge aspect of the whole push for moving from teaching to learning. I think it had that big of an impact.

Rebecca: In recent months ChatGPT has hit the news everywhere. We’ve heard all kinds of reports in higher education journals and things. accreditation organizations are talking about AI. And many faculty members are also talking about potential harms of AI or other see benefits. What do you think the impact of AI tools will be on higher education practices? …speaking of evolution… [LAUGHTER]

Todd: I don’t have the statistic right in front of me, but there was one I did a presentation just recently on how long it took to get to a million users. And they were talking about some things like Netflix and other things of taking two and a half years, and then six months and stuff. And I believe ChatGPT was five days. So it’s pretty incredible. So Rebecca, I love the question. I think this is a huge issue. The two biggest things that I heard out of the gate was: number one is how are we going to ever have students do their own work again? It was all academic integrity all over the place. And I couldn’t help but talk to a couple of people who were really freaking out about it and I said, “You should really read some of the research about academic integrity already and the proportion of students who are doing things that they shouldn’t be doing, so to speak, there.” And so that was it… academic integrity. And number two was like, “What kind of assignments should we do now? How do we structure assignments to be meaningful?” And I find this fascinating, because they weren’t talking about the learning process. And I still, to this very moment, I’m just starting to hear it now, but I have not heard much of anything about how can we use generative AI to influence the learning process in a positive way. And so I think that’s the huge thing we need to be looking at. So I guess the people who do know me, know me as the individual who writes a lot about helping students to learn and teaching tips in the classrooms kind of things. I think from a cognitive psychologist’s point of view, we can look at some of the foundational things like: we know that repetition is really good, we know that the process of explaining something really helps because it’s a chance for you to practice at retrieving information. We know spaced recall is really good. And there’s all these things out there that can kind of do that. But I’ll tell you, and I’m coming right back to the CATs, but ChatGPT and some of the other generative AI has some phenomenal opportunities. We know that teaching this is good. So I opened up by ChatGPT 4 the other day, and I was practicing and I typed in: I need to learn about metacognition, would it be okay if I taught you metacognition, and then you tell me how good of a teacher I am. And the chatGPT says, “Yeah, this will be fun. So let’s get started.” And I said, “Okay,” and so I explained metacognition. And ChatGPT asked me some questions like, “Well, what does this mean? And what does that mean? …and I acted just like I was teaching somebody. And then when I was all done, I said, “If I were a teacher, right now, how would you grade me?” And I did not get a good grade [LAUGHTER] which was kind of disconcerting there. But I said, “Why did I not get a good grade on this? And then it explained a section that I had completely forgotten, because I was doing this off the top of my head to try it. So I went back and explained that. And then ChatGPT says, “That’s much better.” I think that’s phenomenal. Another one, I will not get into details, but I typed in, or I gave it the prompt… it’s all about the prompts folks… “That I just learned in my introductory psychology class at a medium-sized, midwestern university. I’m a C-level student who’s worked really hard, and what I just learned were about persuasive techniques. Could we pretend we’re on a car lot, and I’m a salesperson, and you’re going to buy a new car, and you think you want to buy it today, but you’re not going to rush into anything.” And ChatGPT says, “This sounds like fun. Tell me about your best selling model.” [LAUGHTER] And I proceeded to explain all these things. And the scariest one is I said, “This Ford van right here has excellent ratings for safety.” And it said, “That’s particularly important for someone with a loved one and two furry friends.” So the learning possibilities are all over the place for this thing. And so for classroom assessment techniques, this is great, because we can actually teach students some strategies, how in the classroom, how to assess whether they’re learning. We can replicate these and have them do them with AI. So we could use AI to do things we couldn’t do before. But we can also do it to model things for students that they can do on their own. So I think there’s some really, really interesting possibilities coming along.

John: And one interesting one is if you ask ChatGPT for your biography, you may very well turn out to be a car dealer in the Midwest.

Todd: You know what, I’m just gonna jump in there and say that is totally true when ChatGPT was about six months old, because I did it. I said, “Write a letter of recommendation for Todd Zakrajsek… and you probably saw this, why you even mentioned it… and I think it had 19 things I counted in there and this term of hallucinations, but 17 of the 19 things were wrong, including where I went to school where I worked and everything else. That was when ChatGPT was a baby. I asked it the same thing just recently, and it was scary, how close it just nailed everything. But yeah, it was for a while.

John: It’s improved quite a bit, but one of the nice things about it is our students come in with really diverse backgrounds and we can provide supportive materials for students when they have some gaps in their prior learning. But, ChatGPT offers the possibility of having something that’s completely personalized to their needs, so that they can ask questions that are specific to the issues that they’re having trouble with, and not what we might have guessed they’d be having trouble with. So, it’s something I’ve encouraged my students to use in my intro economics course, and they appreciated the fact that they were encouraged to use it,

Todd: I think that individualized instruction is going to take on a whole new level. And so I think that’s going to go crazy. But I will say this, and I think it’s extremely important. I’ve been in higher education for 40 years, I have never seen a bigger possibility for an equity gap to be the widest chasm it could ever be. I think we got equity problems that are down the road. And if we don’t pay attention, it’s going to be awful, because I believe that generative AI is going to be a huge learning aid, and it’s going to help us with lots of things, for the people who have it available to them, they will skyrocket and be able to do stuff; the people who don’t have it available… reliable internet, a good computer, or some kind of a device, a safe place to sleep at night… those individuals are not going to be on that tract and they’re gonna get separated. Right now, there’s a couple of grade differences between people who are privileged and not in some classes. That could widen so much. So I think, everybody, we just got to really be careful about how to make sure that this was as equitable as possible for everybody.

Rebecca: Higher ed leadership at various colleges and universities are certainly dancing around the idea of AI and what kinds of policies and procedures to put into place. What would you advocate for? What would you encourage faculty members to advocate for to really make sure that equity is addressed?

Todd: I think one of the very first things we need to do is teach students how to use it properly. Because one thing we know about learning is, the more you know about something, the easier it is to learn something related to that topic, whatever the thing is, and what that means is that learning is not linear, it’s curvilinear. The more you know, the easier it is to learn something, which means you’re going to know more quickly, and as you know more quickly, you’re going to be able to learn more quickly. And this is why people struggle at the beginning with things and then all of a sudden they take off. So if some students know how to use the systems, and some don’t, that’s where we’re going to run into some huge issues. So I think one of the things we can do is teach students how to use it to help you, and when to be careful that it’s not going to help you. The movie Wall-E is a great thing to be keeping in mind. If you haven’t seen the movie Wall-E, it’s a plug for the movie Wall-E, it came out a long time ago, but that concept of the humans that had everything done for them ended up becoming just blobs that laid in chairs all the time, and I think our brains basically do that. If students use AI to do their work, they’re not going to develop critical thinking skills. If students use AI to help them to practice at retrieval and spaced out practice and do those things, they could become very good. So I think schools need to be careful about helping students to understand how this can really help them. That’s the first thing. Then there’s the obvious, helping people to have access to technology, we could have labs open, we could have laptops that we can loan to people, reliable internet, we could have rooms that we keep open 24 hours a day, keep them 24/7 there. Some libraries during exam time, they don’t close for like four or five days. We could have rooms in the library that people can come and have safe spaces to study and work, reliable transportation is going to be an issue, we’re just gonna have to work out how do we address those issues. It’s gonna be a challenge, but we need to be thinking about it.

John: The use of classroom assessment techniques has been growing steadily since they were first introduced. And each time there’s been new educational technology, it seems to have led to an increased use of those. During that time, we’ve seen the introduction of relatively low-cost computers, computer networks, the internet, mobile computers, and so forth. How do you think the availability of generative AI will affect the use and value of classroom assessment techniques.

Todd: I think there’s a couple things that’s going to really, really change. I think for classroom assessment techniques, number one is we’re going to be able to individualize a little bit more. So we can tailor… we don’t have to ask the same question of all the students, we don’t have to look for all the same type of responses. So we can think about it a little more creatively in how we can use it, almost like an individualized instruction toward classroom assessment. Probably the biggest thing, though, overall, is it’s going to allow us to just process data in numbers and levels we’ve never seen before, and particularly free responses. In the past, we’ve often ask closed-ended questions just because if you’ve got a class with 400 students, you can say: “On a scale of one to five, to what extent did today’s lecture help you to understand something?” And the students can pick a number, we could drop that into an excel sheet, or anything, and come up with a number very quickly. We can now say on a muddiest point, “What are you still struggling with?” Take 400 responses, dump them into a generative AI program and have it spit out five things within 30 seconds:” Here are the five things your students are struggling with.” So I think it’s going to allow us to do more qualitative types of things very quickly. And I totally get that this is not hardcore qualitative research with good analysis of the data. But for what I need in the classroom, I think we’re gonna be able to get those responses very quickly and in real time. So what that means is, we’re actually going to be able to ask something like a muddiest point at halfway through the class, and then I could have the students do a quick think-pair-share, while I analyze the information and a matter of three minutes later, they come out of the think-pair-share, to talk a little bit about what they talked about. And then I could say, I see that you’re still struggling with these concepts, so let’s revisit these things before they ever leave the room. That’s never been possible before.

John: When I’ve been teaching classes of two to 400 students, I’ve used the free response option and word cloud. But the word cloud is just highlighting individual words, the ability to do analysis in more detail is going to be incredible. And I think most providers of response software are working on introducing AI, and some of it is expected to be available fairly soon.

Todd: Oh, there’s some great stuff come check GPT and some terribly scary stuff at the same time. By the way before I forget this, because I have to tell you this because I have ADD, that’s one of my favorite phrases.ChatGPT 5,, one of the things I read about that’s going to be coming up fairly soon is the ability for it to launch his own AI’s as needed. That’s the one that we’ve been waiting for, and thinking, “Hmmm, that’ll be interesting.” But yeah, I think that the ability to read through and look through the information is just going to be a game changer. And online synchronous/asynchronous, face-to-face in all environments. We could look at stuff in asynchronous environments in ways we haven’t before as well. I could look at different ways that students are responding, I could ask classroom assessment techniques. In fact, they’re in the book. We have several of them… ways to use CATs in asynchronous environments, and AI is changing how we’re doing it.

Rebecca: Can you give an example of how that’s happening in asynchronous environments?

Todd: In terms of asynchronous environments, everybody likes to go to discussion boards, but quite frankly, discussion boards are boring. We could have students generate something like the script for a commercial. And after they do that, I could have the CAT that comes in to say, “To what extent was this helpful in the learning process for you?” And the student, as they’re developing this thing and as submitting it they could also submit, right along with it, their CAT that comes with it, and then I could be reading the CATs as they come in one by one right behind each one of them. And so the concept, there, would be kind of a real time CAT analysis that’s not waiting until there’s a whole group and then looking at the group. Because typically, when we think about these CATs, it’s like, the 400 students fill out a muddiest point and we analyze the 400 students. We don’t need to do that. In an asynchronous environment, we can have these CATs coming in and we can be analyzing them as they come in. And we could even add them to the previous ones, to group them if we wanted to. But the bigger one is, I can find out how the students are doing as they’re doing. So that’s probably one way to look at it.

Rebecca: One of the things that you mentioned in an in-person or a synchronous setting was being able to analyze results mid class. And that’s in part because a lot of our students have mobile devices and technology that they bring with them that they haven’t had before. Do you see other ways that these mobile devices are also changing the way instructors might implement CATs?

Todd: The way you just mentioned, though, I can’t think of many other than the concept of as an information gathering device. I will say, just because I happened to read this study just a little bit ago here, the devices are very interesting, in a sense that we’ve seen high schoolers tend to have the same amount of time they spent with their peers was very consistent until about 2012. And in 2012, the numbers started dropping off pretty quickly. And then COVID dropped it even faster, but the slope was already there. And the point here is that people are turning their attention to devices so much that they’re not talking to each other exhibited by when you see two people at a restaurant having dinner, and each one of them are on their device. There’s all kinds of psychology about how those things happen. But the overarching thing that’s the issue is that we’re having students at levels we’ve never seen before disengaged in the classroom, even when we’re trying to do CATs and engage. And part of that is because of the devices. So one of the things that’s kind of interesting about this is when you say: “Okay, everybody gets your devices out, and now we’re going to use them for a CAT.” And then we stick with that and how we use that data makes a big difference of whether they stick with it or go back to what they were doing. Strategically, it’s really helpful to use those devices to engage the students.

John: One concern with generative AI is many of the types of assessments we’ve used before can be answered very nicely by generative AI. What might we do to reduce the likelihood that students will use generative AI as a substitute for learning?

Todd: I think it’s really, really important to talk to students about the long-term implications. I know this is not going to be for everybody. I totally get that. But one thing I’d say to students is: “I don’t understand why anybody would run, why would anybody go jogging? You can just get in your car and get there faster.” So if you kind of pitch that to them, that idea is for most of the students, if there are any kind of health-related fields, or if they want to have a cardiac system later in life, exercise is just one of the most important things you can do for your body, by the way, in terms of that getting 150 minutes per week, and just higher respiration. If you take that away, it is bad for the human body. And so the response is the same thing is true of this. Just talking to the students. If you use generative AI to answer your homework problems, to develop the quizzes, to write a poem, and I’ve done all of these in workshops, where I’ve said to people, I can do this real quickly. I don’t care who’s in this room, I can ask you to write a short story. And I will crush you with generative AI. The problem is that, if I do that, I don’t learn how to think for myself. And so I think the biggest thing we can do for students, number one is to build community in the classroom, to number two, to tell them what this all means, and then number three, ask them, ask them, how it’s helping them, ask them how it’s hurting them, how they believe these things are working. And those are classroom assessment techniques, we can use these assessment techniques to find out to the extent that they’re doing this. While I’m on my little diatribe, we will never be at this spot again in our lives. We have just developed something that can totally short circuit cognitive processing and critical thinking. But we did it with people who have critical thinking skills and cognitive processing. So we developed something because earlier in our lives, we had to learn in the ways of joggers, we had to develop our systems. And now we can talk about the automobiles that would move as faster. The students coming along are going to step right into those automobiles, right into the AI, and if they’re never told that it’s important to go out jogging, they’re not going to develop the systems that are going to be needed later. And so again, just think about this for a second, we’ve developed a system using a system that may disappear if we’re not careful. And so I think that’s where we can change it. So it boils down to community, why we’re using it, and then assessment questions of how it can be used, and where it’s good and bad.

Rebecca: I think students really enjoy having conversations about AI and exploring how it’s useful and not. I have had activities in my class where we intentionally used AI to see what it was like and when it would be useful and also analyze where maybe they tried to use it and it was totally not helpful, and why it wasn’t. And they really appreciated those kinds of conversations and learned a lot from those.

Todd: I think the students do. And we could pick different spots in history. You go back to Socrates and his whole belief that if you write things down, it weakens the mind. It makes sense. If you have to learn it, you’re going to be much stronger than if you write it down. But I can’t imagine right now teaching without students writing things down. And Samuel Johnson came along several 100 years ago and said, “There’s ready availability of books. With books all around, why would we really need to teach this stuff, they can just go get a book.” I can’t imagine teaching without books right now. So these things that everybody got scared about, or thought that just going to change just became integrated. And when the internet came about, we talked about this a while back, in the sense that many of us were teaching before the internet actually showed up. But when it showed up the people who were teaching were freaked out. “How am I supposed to teach when the students can go and get anything they want off their computer?” And now I can’t imagine teaching without the internet. I believe five years from now people are going to say I can’t imagine teaching without writing things down, without using some kinds of print format, without using the internet, and without using generative AI. I don’t even know how I would teach without it. What that’s going to need is the same stuff we’ve done in the past, that’s how do you teach well with the Internet? How do you teach well with generative AI?

John: It’s an exciting time to be in the midst of all these changes. And it’s going to be interesting to see how we all answer those questions as we move forward.

Todd: It is, but before we wrap up, because I can tell that tone, [LAUGHTER] I want to put a general call out that there are several people who are saying: “I wish I could retire right now” or “This is a great time to retire.” The statement I really want to make is this is a hideous time to retire if you’re really good at what you’re doing. Because we have never needed humans with really good critical thinking skills as much as we do now, maybe we will later, but to date I don’t believe we ever have. And for some of the people who are saying “I just don’t want to teach anymore. This is awful…” we might need you more than we’ve ever needed you, so I don’t like this concept of the mass exodus of certain people.

Rebecca: It’s an important point.

John: One other thing I will add is Paul Samuelson once said, in describing the way in which models of the economy evolved, that “funeral by funeral, the science makes progress.” So there is the counter argument there too, that people who are very tied into the old ways of teaching do need to either adapt or perhaps they’ll be replaced by people who are more willing to try new alternatives.

Todd: Do I smell a little Kuhn in there? The revolution of science. Ah yes.

Rebecca: And we always wrap up by asking: “What’s next?”

Todd: I’m trying to keep busy. I’m kind of on a roll right now. And I want to tell you, I’ve been writing books and the only reason I’m unhappy about ChatGPT and gen-AI come along is people are going to assume that I’m using them so that I can crank stuff out. But thus far, I have not used any of this generative AI to write any of these last books that I’ve done, these five books in the last five years or something, and enjoyed working with Tom on this one. But I got another one coming out in about four months. I just turned in the final pages for that And it’s the Essentials of the New Science of Learning: the power of learning in harmony with your brain. That book should be out in a couple of months. And so the what’s next after that, I am working on a book right now for helping with neurodivergent learners. We’ve joked around in the past, I’ve got ADHD is about as bad as it gets. If it were a competition, I could come in probably close to first place, and also on the autism spectrum just a little bit too. And when I started sitting down with some of my colleagues who also have ADHD and autism spectrum, it occurred to me, we don’t talk to students enough about these things. And I think the challenge of that is that the students believe that they’re alone, and they’re not sure they can do things. And I was talking to a student fairly recently that said, “I now believe I have a shot, here you are with a PhD, you’ve written books, if you’ve got ADHD, as bad as you claim you do, maybe I can do something.” And then that’s when I really really was getting serious about we need to help folks out. And the topic I’ve been playing around with a little bit in a couple of workshops is The Ones Too Often Left Behind as the title I’m using. And it’s the students that aren’t built for the system that we developed. And so I think that we need to treat some folks out there a little differently. And I think we can really build up the pool of intelligent folks by helping to teach the people who just learn in a different way.

Rebecca: It’s really important work to have models and put models out in front of students because they need to see themselves in whatever discipline, field, etc., that they want to pursue. I’m excited to hear more about that.

John: And when will these books be out? You mentioned the timeframe for one of them.

Todd: The Essentials for the New Science of Learning should be out in September. And then the other one, if everything goes well, is probably looking at a February date that it would probably be available.

Rebecca: Well, it sounds like we’ll be talking to you soon then, Todd.

Todd: We may be. When you got ADHD pretty bad, you just can’t really predict when that books gonna be. But yeah, I’d love to chat with you when it does finally emerge.

John: And what are you going to do next week?

Todd: You know what? You gave me a great idea. I think we should do a book on procrastinating people with ADHD. [LAUGHTER]

John: It’s always great talking to you. And we’re looking forward to each of these projects coming to fruition and we will be talking to you about each of them, we’re hoping.

Todd: I appreciate the opportunity to come and chat with you. I think you two do a phenomenal job with your interviews and the programs that you pull together on really good topics and I just am honored to be one of the guests.

John: Thank you.

Rebecca: Yeah, thanks. We always enjoy talking to you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC] </p

333. High Structure STEM Classes

Posted in active learning, authentic learning, clickers, edtech, equity, evidence-based practices, higher education, inclusion, learning, pedagogy, scholarship of teaching and learning, self-regulated learning, STEM, teaching, two-stage exam
March 20, 2024

Multiple studies have found that increasing course structure reduces equity gaps and provides benefits to all students. In this episode, Justin Shaffer joins us to discuss several ways to increase structure in STEM classes.

Justin is the Associate Dean for Undergraduate Studies and a Teaching Professor in Chemical and Biological Engineering and in Quantitative Biosciences and Engineering at the Colorado School of Mines. He has taught a variety of both small and large STEM classes in multiple modalities using evidence-based approaches and has won multiple teaching awards as a result of this work. Justin is also an active researcher with 16 peer-reviewed publications and serves as the editor for four STEM education journals. He is the author of a forthcoming book on high-structure course design coming in late 2024 or early 2025 from Macmillan.

Show Notes

Transcript

John: Multiple studies have found that increasing course structure reduces equity gaps and provides benefits to all students. In this episode, we examine several ways to increase structure in STEM classes.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca:Our guest today is Justin Shaffer. Justin is the Associate Dean for Undergraduate Studies and a Teaching Professor in Chemical and Biological Engineering and in Quantitative Biosciences and Engineering at the Colorado School of Mines. He has taught a variety of both small and large STEM classes in multiple modalities using evidence-based approaches and has won multiple teaching awards as a result of this work. Justin is also an active researcher with 16 peer-reviewed publications and serves as the editor for four STEM education journals. He is the author of a forthcoming book on high-structure course design coming in late 2024 or early 2025 from Macmillan. Welcome, Justin.

Justin: Thanks for having me. I’m really excited to talk to you today.

John: We met at the POD conference back in November, and we talked about doing this but we finally got around to actually scheduling it. So thank you for joining us.

Justin: Thank you, John. Yeah, I saw you sitting there at a table by yourself. I knew your face from LinkedIn and I built up the courage to introduce myself and I’m glad I did.

John: Today’s teas are:… Justin, are you drinking any tea?

Justin: I am drinking some tea. I got prepared. It’s from a small little coffee/tea shop in West Middlesex, Pennsylvania. I’m from the near that area in western Pennsylvania. We call it the Shenango Valley and it’s called O’Neill’s and they make lovely Western PA coffees and teas and I just have some nice black tea here. It is that time of day for me that I need some.

Rebecca: Today I have a jasmine green tea, John.

John: Very good, and today I have a Prince of Wales tea, and a peppermint tea because I didn’t have very much to drink today, and my throat was getting really dry in my econometrics class. So I wanted to be well hydrated. [LAUGHTER]

Rebecca: Lots of variety, too. So we’ve invited you here today, Justin, to talk about how you’ve implemented high-structure course design. And for listeners who might be new to the concept, can you talk a little bit about what you mean by high-structure course design?

Justin: Yeah, so just like I’ll tell you all like I tell my students, I didn’t come up with this term. I wish I did. But I’m building off the work of other giants in the field. So as far as I know, the term really came out in the mid-2000s, 5-, 6-, 7-ish range from the University of Washington in Seattle. There’s some pioneering folks, Scott Freeman and Mary Pat Wenderoth in biology and they use this term structure to think about, “Well, how do we scaffold student learning before class, during class, after class, and we’re continuously building as we go up through the three levels.” So pre-class involves some kind of content acquisition, that can be reading your textbook, reading journal articles, watching Khan Academy videos, other sources like that. And then some kind of formative assessment before class because we all know students need a little bit of a carrot there to get some of that work done sometimes, which is totally great, because you give them a lot of chances, sometimes even unlimited attempts… again, formative assessment before class, make sure everyone has the same solid foundation, then you come to class, high-structure course design, just like any good class nowadays should be really rooted in active learning. So we got a lot of activities going on: group work, problem solving, case studies, real-world connections, any kind of flavor you want to do there with your active learning as long as students are engaged. And the big thing here though, is that you’re aligned to what happens before a class. We’re building off what happens before a class, I’m not in class just repeating what happens ahead of time. Rather, I’m using the foundational material students acquire before class, we build that up with another level in class, and then we keep going. And then after class, there’s a bucket there, where students are going to be practicing those skill sets they’ve been developing through the before class and in-class buckets, applying their skills to new contexts, solving new problems, doing projects, essays, more authentic assignments, whatever you want it to be. And then there’s typically a frequent assessment piece with a high-structure course design too, whether that’s a multiple midterm model, weekly quizzes, every other weekly quizzes, things like that, in addition to some more authentic assessments, which is something I’ve tried to get more into this past year, by adding more realistic projects and course assignments in my high-structure courses.

John: Could you give us some examples of some of the authentic assessments that you use in your classes?

Justin: Yeah, thanks, John. So this is really new to me still, this past year. I know I’m behind the curve. I know this is a term has been around there for a while. I always like to think I’ve had some authenticity in my classes for a long time. I integrate news stories, I use case studies, I have this new thing I called pod cases where I take podcasts and combine them with case studies and we use those in the classroom. But I thought I could try to be a little bit more authentic, even still… getting that advice from my wife for my personal life. She always says that can be more authentic. So, maybe I can make my courses and my assessments more authentic too. One example I’m doing right now. I’m teaching introduction to biomedical engineering, and I’m having the students write the actual legit National Science Foundation Graduate Research Fellowship proposal. Those of you familiar with that, there’s a personal statement part and there’s a technical kind of research proposal part. So we’re just doing that. It’s a two-page proposal, and we’re working on this in a structured way throughout the entire semester. Last week, students had to turn in the research topic and research question. This week, they’re turning in their hypothesis, in a couple weeks they’re turning in the literature review and experimental plan. We’re gonna have two rounds of a formal peer review of the entire proposal. And then we’re going to conclude with a mock panel discussion, just like you might have in the real NSF or the real NIH where students are going to be reviewing each other’s proposals and basically pitching them to each other and seeing if they should get, quote, unquote, funded or not. I don’t have any funds for them. Maybe we’ll bring candy on the last couple of days. But that’s kind of the idea of having a real thing that students could actually use some day if they’re going to graduate school. But even if they’re not, being able to write technically and persuasively in two pages on a proposal is a really important skill that they should be working on, in my opinion.

Rebecca: Sounds like you’re going to need some chocolate coins for that day.

Justin: That’s a great idea. I will get chocolate coins. [LAUGHTER] I didn’t think of that. [LAUGHTER]

Rebecca: Can you talk a little bit about what motivated you to attempt and move in the direction of high-structured course design.

Justin: I kind of started out this way, I kind of started in the deep end, if you will. So my postdoctoral program was called spire, S-P-I-R-E, and it’s one of these bigger programs called IRACDA through the NIH. They’re postdocs where you still do research. So I was a molecular biologist, biochemist at the time. I have a weird background in chemical engineering, bioengineering, molecular biology, but at the time, I was studying muscle proteins in squid, super fun stuff. And then I also got training, though, on how to teach evidence-based college science and engineering courses. In my postdoc, through great people like Ed Neil, Brian Ray Bozic, Leslie Lauria, and then also Kelly Hogan was someone that took me under her wing at the time, she was able to, along with the rest of the SPIRE team, helped me be introduced to using evidence-based course design practices, which included with high structure. So this was around 2010-11, right when these papers are starting to come out from Scott Freeman and Mary Pat Wenderoth. I started with high structure because the evidence shows it works. And that’s as a trained scientist, as a trained engineer, we like to go with evidence ideally to guide our decisions. And it’s no different with course design and teaching as it would be with thermodynamics or reactor design or anything else. The literature on high structure is really strong, even then it’s been growing over the years, some of my own data and a lot of other papers. The main gist, though, is that by having the scaffolding before class, in class, after class, students are able to really monitor their learning, develop self-regulated learning skills, be more metacognitive. And the data in a lot of papers shows all students are doing better in the class. But it’s not just everyone, using high structure and moving up in structure will start to close or reduce achievement gaps or performance gaps between historically underperforming students and their majority counterparts. So for example, female students and male students in physics, there has traditionally been a gap there, add some more structure to the class, those gaps tend to get smaller. And I’ve been participating in that kind of research here over the last many years with undergraduate students and my research team. And we found some similar results across different STEM disciplines. It’s really the evidence guiding me in these decisions to do it. I also think it’s a lot more fun. Once I have in mind one of my very first faculty interviews, oh a long time ago, now, I had an interviewer say to me, “Well, hey, I’ve looked at all the papers on active learning, and they’re all wrong. [LAUGHTER] All the statistics are wrong. What do you have to say about that?” And I paused for a second I said, “Well, even if that’s true, it’s a heck of a lot more fun to teach actively with active learning than not, I think the high-structure design model is more of a fun way to teach. And again, you’re helping the students with those study skills and metacognitive skills that they can then take forward to their future classes.”

John: There was a recent paper that did argue that many of the studies on active learning were flawed in some way in terms of a very strict methodological approach. However, there are so many of them, and the results are so remarkably consistent. I don’t really buy that argument in terms of the questions about methodology.

Justin: You’re right, it was a big deal, I think it was either Beckie Supiano or Beth McMurtrie in the Chronicle had articles on this going around for sure. And that’s the thing though, with us, as DBER folks… I want to introduce a new term, DBER – discipline-based education researchers… which I consider myself to be in. So I’m in the discipline. I’m in chemical engineering, I’m in biology, I’m in Biomedical Engineering, and I do the education research from that lens, but also from that training background. I’ve never been formally trained in education research methods in psychology or social sciences or psychometrics. I am, totally honest, I fake it. I do what I can. I know my boundaries. I know when I need a collaborator in the true social science disciplines that helped me with my studies. And so I think that paper was getting some of that. Sometimes the folks like us that are doing this work, whether we call ourselves DBER people, or Scholarship of Teaching and Learning (SOTL) people, we might not have that true background, and therefore the rigor of the paper, the methodologies, aren’t as advanced as maybe they should be. But like you said, there’s so many studies showing benefits of this, whether it’s simple correlational studies… sometimes they get to the mixed methods, even quasi-experimental studies, but the evidence just keeps mounting and mounting and mounting. And that key paper there was the 2014 one from Scott Freeman that PNAS meta-analysis showing that hundreds of papers, hundreds of studies… again, they have their flaws, don’t get me wrong… but you start to see these themes everywhere and you think, well, there must be an effect, something’s going on.

John: And we will include a link to that paper in the show notes, if anyone would like to follow up. But the fact that there may be these flaws in some of the research methodology suggests that maybe we need to see more people doing work in this area, and that there’s a lot of areas where there’s potential growth.

Justin: Totally agree.

John: And you’ve done quite a bit of research already.

Justin: I’ve tried, as a teaching faculty member now for this is going on year 12-ish, but I’ve always made that a part of my career. Before I was at Mines, I was at University of California, Irvine, where we did have a scholarship expectation or to earn tenure as teaching faculty. The UC system is one of the few in the country where you have actually tenure-stream teaching faculty, which was really cool. So I earned that, but I only had tenure about a month because then I left and came to Colorado, which… no regrets, I love it here at Mines… but I don’t have tenure here. But there’s still some expectations for scholarship, but not as high as it was in the California system. But I continue to make that part of my career, because, again, as a scientist and engineer, I think going based on data, based on evidence to guide your decisions, moving forward is a really important thing to improve what you have, because that’s all it comes down to really whether you call it SOTL or DBER or whatever, we’re trying to improve our courses, we’re trying to improve our students’ experiences. And yeah, so I’ve tried to do what I can over the years. And now I’m at the point where I want to help other people too. So I work with faculty individually, I do workshops via my side business, and even at Mines this semester, starting next week, I have a three-part getting started in DBER series kicking off for about 25 faculty on campus. I’m going to help them get started, get their feet wet in this field, and then we’ll see where it goes. Because I think a lot of folks, like you said, John, want to do this work, they’re interested in it, they might not have the time, they might not know how to, but you kind of give them the start, hold their hand a little bit and then see where it goes, and we’ll get even more evidence to support these claims.

Rebecca: I think a lot of faculty have an inclination to have a reflective practice and continuous improvement. They’re wanting to do SOTL work, but just don’t know how to formalize that practice. And we can certainly lean into helping people move in that direction.

Justin: 100% Rebecca, even when I started, I didn’t know what I was doing. If it wasn’t for Kelly Hogan and a few other folks at Carolina and then moving on from there at UC Irvine folks like Brian Sato, Diane O’Dowd, Adrianne Williams, they really helped me understand the field more. And then you start going to conferences and going to meetings and learning more about the tricks of the trades.

Rebecca: Can you talk a little bit about ways that a high-structure synchronous course might be different from a high-structure asynchronous course?

Justin: So I think the high structure while what I described earlier, you might be thinking of okay, a Monday, Wednesday, Friday class meeting at two o’clock in a lecture hall works great. I think it works even better in online for some cases, because you think about an online situation, whether again, it’s synchronous or asynchronous, your students you’re not really seeing and there’s that human connection of having eyeball to eyeball being able to understand, “Okay, I’m here I’m motivated, I’m ready to learn.” But whether it’s the recording, that’s the worst case scenario, when it’s totally up to you, or if it’s going to be on that live Zoom remote, where you can make your screen black, and who knows what else you’re doing. I’m guilty of that in meetings for sure. But having the structure though, having that scaffolding, having the clear deliverables of what is due when and seeing that step up from pre-class, in-class, after class, whether it’s the in-class is out again, synchronous Zoom session, or the recorded video you do on your own time, there’s that scaffolding helps students stay more connected with material, they understand more of what’s going on, when it’s due, what is due, how I’m being assessed, because that’s the nice thing about with structure. And just to be clear, I’m from the STEM disciplines, but you could do this any field. The research on high structure, as far as I know, is only been published in the STEM fields. I bet it’s being done in art history, I bet it’s being done in the languages, I just don’t know about it. So if anyone listening does this kind of teaching or research, I’d love to hear from you. But you can use that scaffolding again to help students be more transparent with the course. And that’s a word that I think comes through with high structure and is really true in that online environment, especially as transparency. I always tell my students, I want to know what you know, not what you think I want you to know. And by being uber clear with the learning objectives, starting with backwards design, just a clear Canvas website for the online cases can be really helpful for that. One distinction, though, with the async online versus synchronous online, is that kind of active learning piece, because we’re not there together. So one tip I like to share with faculty is that okay, if you’re recording a video for the in class, mimic, because you’re not actually meeting in class. What I always say is “okay, well make the video, but if there’s a point where you’d want them to answer a clicker question… I’m a big clicker guy… or if you want them to do an activity, just say, Okay, everyone, pause the video here. Take a couple minutes, think about this problem, what’s the answer or work on this, whatever it might be. When you’re ready, unpause the video and keep going. That’s a way that you can add the active learning to an online asynchronous class as part of the bigger high-structure model. I find when I have to record an online video, if I have traveling for a conference because I have to miss a day of class. I’ll have a 50 minute in-person class, when I make the video recording it’s only about 20 to 25 minutes, because all the active learning I don’t do in the video recording, but I give that tip… I tell my students, hey, pause the video here, answer this quick clicker question, solve this problem, come back when you’re ready. And so that’s a way to build the active learning into the asynchronous part, which again, is part of the overall high-structure piece.

John: And also, if you want to take it a little bit beyond that, you could use one of the tools where you can embed the questions directly in the video, and have them auto graded. I use PlayPosit for that, and it’s possible in Panopto, and EdPuzzle is another alternative that many people use. PlayPosit works really well. And I’ve been using it for a couple of years.

Justin: I just learned about PlayPosit maybe three weeks ago, I was doing a workshop at CU Boulder on clickers. And I kept hearing PlayPosit, it does seem like a very powerful tool to automate that active learning in the video piece, which I really like. I just haven’t had a chance to play with it.

Rebecca: So you did just mention that you’re a clicker guy, I believe. So do you use a flipped classroom approach in all of your synchronous classes? And how are you using clickers in your classes?

Justin: Yeah, I think that’s the technical term these days, clicker guy. So, I’ll go with that. I’ve been using them for as long as I’ve been teaching in various platforms. You name the platform, I’ve used it. I’ve primarily been using iClicker, just because that was the platform we had at UC Irvine and the platform we have here at Mines as well. Yes, I love using them. I think it’s really important. And you mentioned the flipped classroom design there, too, Rebecca. So when you think of high structure, you might think, “Oh, it sounds like a flipped classroom.” And it kind of is because we’re having students do stuff ahead of time. In class, they’re very active, but there’s still that structure piece even within class. I guess maybe I haven’t seen a truly flipped class because I think flipped classes, I think in class is like purely work time, purely activities, purely worksheets, homework type assignments, whereas I do a series of clicker questions, discussions before or after numerical problem solving, if it’s more of my engineering classes that I teach in thermodynamics, or material and energy balances, more conceptual sometimes if it’s biology or anatomy and physiology, but I’m breaking up a lot of activities and any talking with clickers. And so the clickers… I love all the modern bells and whistles that they have as that’s the another technical term I use, moving beyond multiple choice. And when I do workshops for faculty with clickers, that’s what I triy to get them to do. Okay, yes, you know, you can do A B C D E, but can you do multiple answers? Can you do target questions or hotspot where you touch the screen? Can you do ranking or ordering? numerical answers? …and I’m trying to get this traction on my own campus too. So even this semester, my first five questions of the semester, which I didn’t do intentionally, but after I wrote them, I realized, oh, I should do this every class now… my first five questions in my introduction to biomedical engineering class, each one was a different question type. So we did multiple choice, we did multiple answers, we did short answer, text entry, we did numerical, and we did target or touch the screen. And when I was done, I asked my students, “Hey, who’s ever used those other question types before?” And maybe four hands went up, I said, “Cool. What class did you do them in?” and they said, “Oh, your class last semester.” I seem to be the one and only that I know of on my campus that’s trying out these other tools, although like I said, I’m trying to change that. So yeah, if you’re using any kind of device like this, any type of ed tech, for that matter, make sure you justify it, justify the cost, and you make use of those bells and whistles in class. Or if it’s out of class, courseware, whatever it might be, just make good use of the tool, and you’ll find your students that are going to be able to buy into it and use it more effectively, and hopefully learn more from it.

John: And one option that we have here is a campus license, which actually is much less costly per student. And that reduces that barrier of students having to pay for it, which has led to a pretty rapid expansion in the use of it on campus.

Rebecca: One of the things that I really like about your example of the five questions is that extra level of scaffolding, of introducing all of the question types right away, and giving everybody an opportunity to practice those question types that you’re going to be using all semester. And so I just wanted to make explicit that other layer of scaffolding that you have built in there.

Justin: Yes, teaching students how to use the tools is very important. Thanks, Rebecca, for mentioning that, because whether it’s the app now on your phone for these clicker devices, or if it was the old physical device, there’s a learning curve with them. And same thing, even if it’s computational tool, or like Desmos, or something for a calc class. Yeah, you got to be able to teach students how to use the technology properly first, before you expect them to use it well. I will say though, with the learning curve of the clickers, when I was at Irvine, the app wasn’t really there yet. So everyone had the physical iClicker remote, and this was the iClicker w, so you could do text entry, and you could do number entry, but you had these up and down arrows, which you had to click to go through the alphabet. So I do the type of clicker question where I like to give four statements on the screen, and I say, “Okay, well tell me if they’re true or false, and you have to click in all four.” So you would click in a string of T’s and F, so it’s like TTFT, or FFFT, whatever it might be. And so when I would do this in intro bio there with 440 students, it sounds like a swarm of locusts, all the clicking to get to the T’s and then someone says, oh, you should change it. Just make A be true and B be false? I said “Great idea, I’ll do that.” But now with the app, we lose the sound effects and iyou can just type in T’s and F’s and it’s a little bit easier to use.

John: One thing I haven’t tried with iClicker, and I do plan to do it this semester is allowing students to rate their confidence in their answers. Have you tried that yet?

Justin: I did just last week. So I just found out that was available this semester, and I gave it a shot. And it was kind of fun. Like the first day I turned it on, they’re like, Oh, look at this, this question was 80% confident, and now it’s down to 40% confident and it kinda reflected the percent answers that were correct. So like the one that was 80% confident it was a single multiple choice answer. And it was pretty much most people got it right. The one that was lower confidence, like 40%, was a multiple select answer. So they had five options, but two or three could have been correct. And so the confidence goes way down through a different question types even. So, I think it was really fun for a day or two, and then kind of the novelty wore off, unfortunately. [LAUGHTER] So I’m going to try to revisit it maybe more in targeted times and targeted questions rather than doing it for every single one. But it was fun to try it. I’d encourage you to do it, too.

John: One of the reasons I was considering it is that it might help students improve their metacognition a little bit by reflecting on how well they thought they knew something and then getting fairly immediate feedback on whether their expectations about how well they knew it actually matched the response, [LAUGHTER] or the feedback to the response.

Justin: Absolutely.

Rebecca: When you’re using clickers, are you also using strategies like think-pair-share as part of your implementation?

Justin: Yeah, absolutely. And there’s that sweet spot of clicker responses that lead to that. So the best practices, I’d say, and a lot of this work came from Carl Wieman Science Education Institute, which I believe is still housed through Vancouver, British Columbia, but he’s at Stanford now. But there’s this great flyer they made like this trifold flyer that kind of walks you through how to use clickers properly, or best practices suggested, and then what are the possible outcomes? Typically, you’d start off with that silence. Okay, click in on your own at first, I tell them I want to hear crickets, just click in by yourself, and then I’m able to see on the app the distribution and so at that point, there’s typically kind of three outcomes. So one is maybe 80-90% have the correct answer. At that point I’ll say “Okay, let’s move on.” But I make sure to take time to have students respond and say, “Okay, well, why is it C?” or to have a little discussion about why it’s not the other answers. Because even though you get 90% correct, there’s still 10% of students who got it wrong. So we want to make sure that everyone’s on the same board. The next option is usually that kind of random distribution. Let’s say if it’s a multiple choice question, you get ABCD each are in the 20 to 30% range. At that point, I call that Hmmm, what’s going on? Was it a really hard question? Did I goof the question? Is there a chemistry exam this evening, so there’s some reason why maybe it didn’t click? And then what I call the Goldilocks case is when you get the kind of 50-50. So maybe you’re split between two answers. And I love actually, when it’s exactly 50-50, I’ll show that on the screen. But sometimes I don’t, and I’ll say “Okay, at this point, now, you’ve all had a chance to try on your own, now talk to each other.” So that’s kind of the think-pair-share that’s built into the clicker there. So this is peer instruction at its best. This was pioneered by Eric Mazur at Harvard, people like Jenny Knight at CU Boulder have studied this. And she actually has a Science paper with Carl and a few others that showed when students are doing this… So you give them some quick information on their own. Don’t show them the answer, have them talk to each other. They put a bunch of microphones around the lecture halls at CU Boulder, and they listened into their conversations. And they actually found students are legitimately teaching each other and debriefing the topic, discussing it, tearing it apart. It’s not just the one student says, “Oh, it’s A ,everyone else, click A.” They’re actually debating it and talking about it, so that you get that true peer teaching and peer instruction going on. And then typically, you see everyone moving more towards the positive correct response that you’re looking for, at which point you can say, “Hey, look, you don’t need me as your instructor, you’re teaching each other. You’re doing great. Kudos, keep up the good work.” Yeah, there’s so many ways to riff on clickers and use them in different respects. Whether it’s the more traditional question types, or now the more advanced question types, and no matter how you do it, you’re getting students more engaged, they’re having more fun with it. And fun actually is a word they use with clickers. I have a paper a few years ago from the HABS educator where I looked at clickers vs. Kahoot!s, and students rated Kahoot! and clickers both equally fun, where Kahoot! being the more gamified version with trophies and a podium and music and sound and stuff like that.

John: And both Eric Mazur and Carl Wieman and some co-authors had done some experiments on the use of clickers with that methodology with answering it individually and then discussing it and voting again. And they found some really substantial learning gains in both of those studies. And we’ll include links to those in the show notes. In all areas, we’ve been seeing some significant equity gaps, but I think since COVID, we’ve been seeing much worse equity gaps in terms of students because learning losses were pretty pronounced everywhere. And that’s especially in math and in the STEM fields, but they were especially large in lower-income communities where schools were not as well funded and where students did not have the equipment and so forth, needed to thrive as effectively in remote instruction. One of the things that some institutions have been using increasingly is co-requisites in place of prerequisites to help students who want to pursue a major in a STEM field, but who would otherwise fall way behind or take many years just to get up to the basic level of math needed for the entry level courses in the discipline. In 2021, though, you had done a study that did a comparison between the effects of prerequisites and co-requisites. Could you talk a little bit about that study and what you found?

Justin: Yeah, absolutely. We saw the same thing probably everyone else did with students struggling, returning from online instruction in high school and having online instruction here. I taught remote synchronous for three full semesters, even though at Mines we stayed open for business pretty much the whole time with in-person classes, just lower capacity. I had prior online instructional experience in my career, so I volunteered to do those online sections for our department. But with the co-rec/prereq issue, I think that’s really important to consider for a lot of reasons. And this work really started when I was at Irvine with Brian Sato and Pavan Kadandale and a few others, looking more in the biological sciences and “do prereqs matter?” It was kind of our big research question. And then I carried that over to Mines and applied that to chemical engineering too. The short answer is: it depends. So first of all, and kind of why do we even have prereqs or co-reqs? A lot of it seems to be more of a management thing. So if we want to control class sizes, moving from first year to second year, and so on, and so forth. But there’s definitely an idea of well conceptual understanding and carry forward for if you have Bio I, well, those skills you develop could be used and count for something when you get to Bio II or same thing with the calc series or whatever it might be. However, we don’t know. It’s unfortunate that I think programs don’t sometimes look at those connections, that vertical alignment, if you will, between courses in terms of prereq to the following class, or with the co-req issues. At Mines, what I did with the paper you’re referencing, John, in chemical engineering, we have these two courses. One is called material and energy balances, MEB, I’ll call it for short. And that’s kind of like our intro bio version for chemical engineering. It’s kind of like a survey course that covers a lot of different topics, gives you some foundational skills that you’re going to need as a chemical engineer. We also have introductory thermodynamics, and an intro thermo, it’s related to MEB, covers some same conceptual understanding and problem-solving skill development, but it’s more on the nature of energy, and that transformation understanding. So typically in chemical engineering curricula, and I’m getting in the weeds here, so apologies for anyone who’s tuning out cuz you’re not a chemie, but MEB comes first, that’s typically either the fall of sophomore or spring of sophomore, you take that. First, it’s kind of like your gateway class, and then that’s a prereq to intro thermo and everything else you take. At Mines though, we do a little bit different. Mines, we do intro thermo first, fall of sophomore year, and then in the spring of sophomore year, you take MEB and coming from that background myself as a student at Penn State, I wish I would have had that way because intro thermo is the more… I use the word gentle… it’s a gentler introduction to chemical engineering than material energy balances is. But the idea here is that you can take thermo as a prereq in the fall or as a co-req in the spring with MEB. So the natural question came up… it’s a nicely well organic experiment to test is, well, if you have the thermo as the prereq do you do better than if you take thermo as a co-req with MEB. And this has implications for advising and its implications for progression through our major through repeating classes, all sorts of things. So it’s a really worthy study and my student, Jordan Lopez, at the time, he was an undergraduate student in mechanical engineering, and I love working with undergraduates on these SOTL and DBER projects. Jordan came along with me for this we did a quantitative assessment to figure out okay, well if students in MEB, did you have thermo as a prereq or co-req? How’d you do in MEB? And then we also asked the students qualitatively “Okay, well, do you think having thermo as a prereq or co-req matters? And if so, what do you think? And what we found was there was actually a big deal. Students who took thermo as the prereq did significantly better in their overall course performance in MEB than if they took thermo as a co-req, and that’s using linear regression models controlling for incoming GPA and things like that. Students also felt the same way. They felt that if you have thermo as a prereq, you would do better than having thermo as the co-req with MEB, and again, the quantitative data bore that out. The reasons they gave for that was because they get some opportunity to get more familiar with the discipline, with how we approach chemical engineering, how we do problem-solving skills, and having that foundation in the prior class allows them to do better in the follow up MEB course. In this case, it mattered, prereq vs. co-req. However, in our studies at Irvine, with Brian and Pavan and others, we found it didn’t really matter so much whether you took for example, microbiology lecture before after microbiology lab, you would think those two would be highly correlated one as a prereq to the other, it didn’t really matter in the result. Same thing with anatomy and physiology sequences, it didn’t seem to matter too much, in other papers in which we cite in these studies looking at organic chemistry sequences, economic sequences, it’s kind of hit or miss with whether prereqs matter or not. Long-winded segment here, but I think we as programs should be taking careful look at our prereq sequences and core sequences in general to see what, if anything, our students carrying over from one course to the other, and maybe we should loosen that up a little bit, because if they don’t matter, that can help students be a little bit more flexible with their degree planning, and time to graduation and things like that.

Rebecca: Speaking of prerequisites, faculty are often complaining that students don’t necessarily recall [LAUGHTER] information from prerequisites as they go into a sequence like we might expect. In 2018, you conducted a study that investigated the effect of physiology prerequisites on the future anatomy and physiology courses. Can you talk a little bit about that study and what you found there?

Justin: Yeah, so that was what I was alluding to there, Rebecca, with some of the work I did with Brian and Pavan. And in those cases, there wasn’t too strong of an effect, if anything, with the sequencing and the ordering that students took these classes. At Irvine, we did have human physiology as a prereq to human anatomy. I know that sounds a little funny to those of you in the A&P space. But the reason we did that was because of kind of a controlling the size of things. Our physiology course there was a lecture-based course, it would have 3-4 hundred students in it every quarter… talking about over 1000 students a year taking this class, versus the anatomy class, which had a lab, we can only handle about 144 per quarter, so availability of seats in the human anatomy was much, much lower than physiology, and also not as many students wanted to take anatomy versus physiology. So that’s kind of why that ordering existed. And so we tested that, though, looking at with what Brian and Pavan and I came up with, we call it a familiarity scale. How do you do on questions that are based on how familiar you are with the content based on the prereq? What we did was we had the prereq instructors look at the follow up course exam questions and write them as very familiar or not familiar based on basically whether it was covered or not. And then we looked at student performance on those questions based on having the prereq or not, and we didn’t see much of a difference based on the very familiar versus not familiar questions. The way we designed those studies, we didn’t come up with any strong conclusions, again, on the biological sciences side saying that, yeah, these prereq sequences matter. But on the chemical engineering side, which I just described, that seemed to totally make a difference, the ordering of the classes.

Rebecca: It seems like it’s really important to think about why we’re structuring the curriculum in the way that we are. And is it just traffic control? Or is there actually a legitimate reason why we’re hoping that they are scaffolded in the discipline?

Justin: Yeah, I think it’s reality, a little bit of both that happens, which makes sense, kind of have that traffic control, but also making sure that that alignment occurs through our new programs or existing programs, and just taking the time as a program or department every so often to say, hey, let’s look back at this and see, why are we doing it this way? Things need to change in this modern day.

John: One thing I wonder a little bit, though, when instructors are complaining about how little students remember from the prerequisite courses that they’ve taken earlier, is how much of that may be due to the incentives provided to students. The way many courses are structured, not in the sense of the structure you were talking about, encourages students to do a lot of cramming before the high-stakes exams. And we know that that results in very little long-term recall. Might some of these issues be resolved if we could encourage students to adopt better learning strategies, and if we could design some of our courses at the lower level, perhaps, to build that in, using the type of structure that you’ve been talking about? Because at least from what I’ve seen, that’s not so common? You picked up this stuff fairly early, but I don’t think that’s part of the training of most of the faculty in STEM fields or in most disciplines.

Justin: Yeah, I agree with that. And there was even in 2018, I believe, in Science, a paper, I think they call it “The Anatomy of STEM Education in North America,” it was something along those lines, where they looked at a big survey a big swath of courses and trying to see who’s using Active Learning who’s not, they use the COPUS protocol, the observation tool to characterize what’s happening in the classroom, and they broke it down in these seven different buckets. And even at that time, a very small percentage of the courses surveyed in STEM were taught using these student-centered strategies, not necessarily high structure, but the active learning, student-centered ideas. But I completely agree with you, John, if we can get more faculty on board with using these types of evidence-based practices, which yes, it takes some time, takes some energy, takes some foresight to develop these courses. But you do the same thing with your research, it takes your time and energy and careful thought to put in your research experiments. We should put the same energy into our classes too. And if we can do that at the early stage, which again, there’s such a huge issue with loss of learning. Not only that, but also lots of students from the early stage. There’s a paper after paper for these bigger courses, taking them as quote unquote gateway classes, we get a lot of attrition at these levels especially of students who might not be as well represented in STEM. We can add high structure. We know that works. We know that helps students do better and succeed. It might not work the first time, don’t get frustrated everyone. There’s a great paper from Anne Casper and Scott Freeman about having True Grit with being able to try and try again. So in her case, she tried adopting high structure at a smaller school in Michigan, it took her about three tries to get the right formula for it to work for her students to see success. And that paper documents that trial and error over a few semesters. But if we can use these types of methods in our classes, ideally, our students are going to be better prepared then for future classes. I do these things called Reading guides, which again, another shout out to Kelly, I got a lot of inspiration from her at Carolina for these, these are just Word documents that I make. And I have a bunch of these on my professional website. If anyone’s interested to download them for free, go for it. They’re just Word documents that help students read the textbook and again be more transparent with what I want them to know. So it has them define terms, fill out tables, make little drawings, they answer checkpoint questions in the text, summarizing things in their own words. And with the reading guides, we’re able to help students again acquire that content and then use it in the class as needed. I’ve had students in follow-up classes say to their instructor, “Well, where the reading guides at?” And they’ll come to me like, “Hey, what are these things? What are they talking about?” So the point being with that story, though, is that I tell students that “Hey, even if you go to a follow-up class, where it’s not highly structured, you’ve developed so many great metacognitive skills, you’ve developed so many great self-regulated learning skills through this class that you can then apply it and add your own structure to follow-up classes, if needed. And these are the kinds of tips that I love working with faculty about to try to help them transform their classes when I visit a campus and do a workshop or work with faculty one on one or on Zoom or whatever, trying to take these practices, which again, are rooted in the evidence, they’re rooted in the literature, but we have to tweak them to your own personal situation, your own student body, your own courses and demographics and everything. You all know your students way better than I do. What I try to do is combine what I know from the literature and the evidence. And that says a lot of this is in the book I’m writing on high structure, it’s going to guide you through the course-design process, a lot of practical outcomes and deliverables. But you got to tweak it and match it with again, your unique circumstances to achieve those positive outcomes.

Rebecca: I think one of the nice things about introducing students to some strategies for high-structured learning and retaining information is that if you have things like your reading guide, and they show it to another instructor, now they might request that of their instructor and the other instructor might decide, “Hey, this is a good idea. This is a strategy I want to adopt too.” And that’s some of the best ways to get some of these techniques to spread.

Justin: It really is. It’s that camaraderie building through seeing each other’s practices, seeing what’s going on. I’ve been part of a project here at Mines since I got here almost six years ago now where we’ve been working on kind of revising how we evaluate, and what we consider effective instruction to be. So we’re trying to take a more holistic review of how we take into account the best practices of teaching and student learning. And one of the big pieces there is peer observation. So we’re actually rolling that out, as of this week and last week with doing departmental trainings and helping faculty see how we can do peer observation to learn from each other. I learned so much from going into another person’s classroom or looking at their Canvas site, or their syllabi or their materials. And I really got this from Malcolm Campbell, who was at Davidson College in North Carolina, when I was a postdoc, this is going back 14 years or so. And Malcolm told me once on the phone, he said, “Find someone who you consider to be a master teacher, and go watch him teach.” But he said “don’t go just once, go a lot. Once a week, if you can for several weeks, and you’ll get to see them handle different situations, you get to see the ups and downs of the semester, you get to see what it’s like on the day where the midterm gets passed back. Or you get to see what it’s like when they’re getting to a really difficult topic or there’s some kind of disruption that class and how they handle it.” That’s the one great thing about peer observation is that if I go to your classes, sure, I’ll be able to give you some comments or feedback on what I saw, but I’m going to learn probably more from seeing you two teach than whatever I could give you and then I’m able to incorporate that into some of my lessons and my teaching style. If we can just change that culture, make it a little bit more okay sometimes to talk about teaching, share teaching strategies, see each other teach and have fun with it really and learn from each other, I think that can go a long ways towards, again, improving student outcomes and helping them do better in all aspects of college and life.

Rebecca: I found one of the most fun opportunities for observations is doing observations outside of your own discipline. It’s really informative to see how people in wildly different disciplines approach teaching and the way that they operate their classroom. I remember visiting a dance instruction classroom when I was at Marymount, at my prior institution, and it was really interesting, it was fascinating to me about how specific and transparent every little move was and how to correct things and the anatomy that was being discussed and just the level of depth that you might not realize how something transpires in a classroom unless you’ve actually had the opportunity to observe it.

Justin: I’m going to play that some for my faculty Rebecca. I thank you, because that’s what we’re trying to do at Mines is not intra-departmental, but rather inter-departmental observation. So getting across the disciplines for sure. Because yeah, you learn so many different things. And even if you’re just going from mechanical to electrical engineering, there’s still a lot of differences between those types of courses, you might see what’s going on. And I’ve learned that myself, I’ve watched some economics instructors teach at Mines, no idea. I feel like I maybe should know this stuff to be a better financially literate person myself, but I learned so many cool ways about how they’re teaching, but I also learned some cool concepts.

John: We do have an open classroom project here on our campus, we keep hoping more people will open their classes and more people will attend. But the people who have participated have found it universally to be valuable. So we’re going to keep doing it, and we just hope to see it become more a part of the culture.

Rebecca: Sometimes just indicating to a faculty member that you’re interested in their content, and you just want to sit in because you’re curious is a great way to learn new content, and maybe get an opportunity to observe someone teaching. [LAUGHTER]

Justin: And yeah, I definitely pitch it that way. Like I want to learn from you. And you get some new ideas. And I feel like in that case, faculty are more than willing to have you in. Well, yeah, Best of luck with that open classroom project. I’ve seen those types of programs in the country kind of sometimes hit sometimes miss. But best of luck with that.

John: One of the things you’ve done is just some research or some experiments involving two-stage exams. Could you talk a little bit about how you implement them, and what you found in general? And how did students react to the two-stage exams?

Justin: Yes, so I’ll try to be brief on these because I could spend probably 45 minutes on these myself. But yeah, two-stage collaborative exams, or more affectionately called group quizzes or group exams for short. There’s been a ton of literature on this Jim Cook, UC San Diego, he really helped me get started in this about five years or so now. But it’s all disciplines. You look it up, and the one paper I have, you mentioned in the intro, there’s a lot of references to different STEM disciplines that use them and some non-STEM too. The idea, though, is that we push active learning, we push group work, we push collaboration, but then we assess you by yourself with a pencil and paper and a calculator on a piece of paper, so why not change the dynamic there a little bit. And the two-stage idea is that first you have students take an individual quiz or individual exam, and I personally have gone to a weekly quiz model with my high-structure courses. I don’t do midterms anymore, I don’t do finals sometimes even. I know that’s taboo sometimes, but it depends on the class and I might not even do that. My colleague in Irvine Adrienne Williams says with high structure by going to the weekly quiz model, instead of having these giant peaks of anxiety with the two or three midterms, we just have a smooth level of anxiety the entire semester, and spread it out a little bit. But, for me, group quizzes then, what they look like is students take the individual quiz, turn it in, then they get in a group, this is a predetermined group, they know who their group is going to be. And then I give them a group quiz. That group quiz in the paper that you referenced was an identical one. So they took the identical two problems. And this was a MEB material energy balance class. So it was just two numerical problems, problem solving. They did that again as a group. However, my other courses now I do a different quiz. It’s either an isomorphic question, so slightly similar, but a little bit different, maybe in intro thermo or if it’s biology or anatomy and physiology, completely different set of questions. Then they do it as a group. The reason we do this, again, was trying to get this collaborative piece in the group setting. But also, when do you find students are most excited to talk about a quiz or an exam? It’s as soon as it’s over, they go walk out, the hallways abuzz, everyone’s all excited, good or bad excited, but they’re talking. So let’s capture that energy and do it in the classroom through that group assignment. Now, the group quiz is usually a fraction of the weight. In the literature, you’ll see, it’s usually let’s say you call the whole experience 100%, the individual quiz, maybe 80%, the group quiz is 20%. But that can vary from 75 up to 95, however you want to do it. And again, they’re covering the same kind of concepts just in that group dynamic. And so when I did this the first time, I got some consultations from my co-instructors, Rachel Morrish, Dave Marr at Mines, and I was a little bit nervous to see how it would go. But it was a hit. It was an absolute blast. The energy in the room is one thing I cannot describe it strongly enough here through audio, it’s just you got to see it. The students are energized, they’re talking, they’re debating. They’re working through the problems together. They’re trying to figure out, I will use the F word here. It is fun. It is a very fun experience to see. I’ve had students tell me that after the quiz is over, like that was a lot of fun. Talking about spreading things on campus. This is one thing that has spread a little bit at least within my department of chemical engineering, a few other classes have been using them now. And I collected some data on this, which I published, showing that… which makes sense, there was no shock… but the group quiz score is higher than the individual score because you’re tackling this together right away. And then also though, students strongly preferred it. The affect, the positive attitude response was through the roof. Both of those results, these are replicated from other disciplines, they see the same kind of things. I think it’s a really nice way to promote that community in the class, show that you’re working together, add that active part to the assessment. However, I’ll end with saying there can be some resistance to this, especially from students who might feel like “hey, I can do it all on my own, I’m fine.” And then they have to work together with maybe some students, they don’t know as well. You might get some resistance from students saying that I don’t want to work with someone else, I wanted this on my own. So be prepared for that. Also, accommodations with testing, if you have students who get extra time, that can be difficult to manage. I will say though, from doing this now about five years and four different classes at Mines, I’ve never had a single issue with students and extra time. What I typically do is I’ll have students get the extra time for the individual part, then they come join us for the group part and have the normal amount of time because in that group setting the students with extra time, they’ve always told me and I always ask them, I say, “are you okay with this?” And they’re like, “Yeah, it’s fine to have the normal time because I’m in a group.” So just some things to be aware of, if you’re going to try them out. And again, I could go on and on with these, so if anyone wants to learn more, hit me up. I’m happy to chat with ya.

Rebecca: Sounds like a good day to visit a classroom, huh?

Justin: It’s an energetic day for sure. Although you might not see much teaching going on if it’s just the assessment part. But it’s fun to drop in and see what’s happening.

John: There is a lot of teaching going on in their second stage, though.

Justin: Absolutely.

John: In fact, when I first did it, I was so excited about it, I made a short video recording and sent it to Rebecca, so she could see the dynamics of that interaction.

Rebecca: And hear it.

John: It was just really, really different than going over a quiz, where some of them would be really happy, but not that focused. And the students who did poorly were generally pretty disengaged when we went over a quiz ift they didn’t do so well. But when they were explaining it to each other, it was so much more positive.

Justin: Yeah. And you’re right, the teaching is happening. It’s coming from the students to each other. Not for me necessarily, but it’s coming from them. And it’s so powerful to see… absolutely to hear that live.

Rebecca: But you’ve set up the structure to facilitate it. So therefore, you’ve done the teaching.

Justin: [LAUGHTER] Thank you. Yeah.

Rebecca: So we always wrap up by asking: “what’s next?”

Justin: Yeah. So for me, what’s next? You mentioned at the top, John, I have this new position, I’m Associate Dean of Undergraduate Studies. And I mentioned in passing, I have a side business I called Recombinant Education. I do workshops for faculty. I’m writing a book. For me, I don’t know what happened, but last year, I had a birthday the ends in zero. Things have been a little wonky in my brain lately. And I listened to Taylor Swift now, sometimes. I never used to, but I think a lot of people do that. So it’s not just me turning that age. But the point being is that I’ve been doing this enough time now that I feel like I have things to share. And I’m trying to do more of that. So I’m trying to, and I’m still nervous to do things like this kind of stick my neck out and even talk to folks like yourselves, but I want to share best practices with teaching and learning, specifically with STEM education, because I still hear from students and I still see it from time to time, students are just not getting the best classroom experience they should be getting. College keeps getting more and more expensive. Students don’t deserve to have less than excellent experiences, in my opinion. I’m doing what I can on my own to share these best practices. Someone called me a one man teaching and learning center, it’s probably a fair way to say it. One of my students said, “Oh, you’re teaching teachers how to teach,” and that’s, I guess, another fair way to say it. But again, I got the energy for it now. I kinda have this midlife crisis going on too where I want to make some bigger impacts on the world outside of my own bubble, whether it’s at the administrative level at Mines now or across the country, or even the world. So I love working with other faculty on all these different areas. And that’s what I’m trying to make a big part of my day-to-day life now. But I’m still in the trenches. I’m still teaching classes every semester, because that really keeps me rooted, lets me try some new things out, still doing the DBER work and the SOTL work, working with students and just trying to have a blast and do what I can here in the time I have left.

John: Well, thank you for joining us. And we’re really looking forward to that book. And I hope you’ll come back and talk to us again when the book is close to coming out. Or sooner.

Justin: I would be honored to come back anytime John and Rebecca and if both of you are ever in the Denver golden area, please let me know. I’d be happy to show you around out here.

Rebecca: Well, thank you so much for joining us. It was fun to talk to you and to hear about the fun happening in your class.

John: More F words.

Rebecca: Yeah, [LAUGHTER] a lot of F words.

Justin: Yes. [LAUGHTER] Thank you both. I appreciate it. It was fun.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

GANESH: Editing assistance by Ganesh.

[MUSIC]

276. Teaching at its Best

Posted in active learning, assessment, attention, authentic learning, backward design, classroom community, clickers, cognitive science, collaborative learning, critical thinking, edtech, educational technology, engagement, equity, evidence-based practices, feedback, first-generation, first-year classes, grading, growth mindset, higher education, inclusion, learning, motivation, online learning, pedagogy, teaching
February 15, 2023

New faculty often start their faculty roles without training in teaching. In this episode Linda Nilson and Todd Zakrajsek join us to talk about the evolving roles and expectations of faculty and explore the new edition of a classic teaching guide.

Now Director Emeritus, Linda was the Founding Director of the Office of Teaching Effectiveness and Innovation at Clemson University. Todd is an Associate Research Professor and Associate Director of the Faculty Development Fellowship in the Department of Family Medicine at the University of North Carolina at Chapel Hill. Linda and Todd are each individually the authors of many superb books on teaching and learning and now have jointly authored a new edition of a classic guide for faculty.

Shownotes

  • Zakrajsek, T. and Nilson, L. B. (2023). Teaching at its best: A research-based resource for college instructors. 5th edition. Jossey-Bass.
  • Nilson, L. B., & Goodson, L. A. (2021). Online teaching at its best: Merging instructional design with teaching and learning research. John Wiley & Sons.
  • Nilson, Linda (2021). Infusing Critical Thinking Into Your Course: A Concrete, Practical Guide. Stylus.
  • McKeachie, W. J. (1978). Teaching tips: A guidebook for the beginning college teacher. DC Heath.
  • POD
  • Betts, K., Miller, M., Tokuhama-Espinosa, T., Shewokis, P., Anderson, A., Borja, C., Galoyan, T., Delaney, B., Eigenauer, J., & Dekker, S. (2019). International report: Neuromyths and evidence-based practices in higher education. Online Learning Consortium: Newburyport, MA.’
  • Padlet
  • Jamboard
  • Eric Mazur
  • Dan Levy
  • Teaching with Zoom – Dan Levy – Tea for Teaching podcast – May 26, 2021

Transcript

John: New faculty often start their faculty roles without training in teaching. In this episode we talk about the evolving roles and expectations of faculty and explore the new edition of a classic teaching guide.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Linda Nilson and Todd Zakrajsek. Now Director Emeritus, Linda was the Founding Director of the Office of Teaching Effectiveness and Innovation at Clemson University. Todd is an Associate Research Professor and Associate Director of the Faculty Development Fellowship in the Department of Family Medicine at the University of North Carolina at Chapel Hill. Linda and Todd are each individually the authors of many superb books on teaching and learning and now jointly have authored another superb book. Welcome back, Linda and Todd.

Linda: Thank you very much.

Todd: Really appreciate the opportunity to be here.

Rebecca: Today’s teas are: … Linda, are you drinking tea?

Linda: I’m drinking a tea called water. It’s rather dull, but I enjoy it.

Rebecca: It’s very pure.

Linda: Yes, very pure. Very pure.

Rebecca: How about you Todd?

Todd: Oh, I’ve got myself a Lemon Detox because I’ve spent most of my day getting all toxed and now I’m getting detoxed. [LAUGHTER] Wait a minute, that sounds bad. [LAUGHTER] But that will be all right. [LAUGHTER]

John: Especially at Family Medicine.

Todd: Well, we can fix it. [LAUGHTER] In general, life is good.

John: I am drinking pineapple green tea.

Rebecca: Oh, that’s a new one for you, John.

John: I’ve had it before, just not recently.

Rebecca: Okay. I’m back to the very old favorite, English afternoon. Because I stopped by the Center for Excellence in Learning and Teaching and grabbed a cup before I came.

John: And we are recording together in the same room, which has been a fairly rare occurrence for the last several years. We’ve invited you here to discuss your joint endeavor on the fifth edition of Teaching at its Best: a Research-Based Resource for College Instructors, that Linda originally developed and now you’ve collaborated on this new edition. How did the collaboration on this edition come about?

Linda: Well, let me talk about that. Because it was pretty much my idea. Jossie-Bass contacted me and said “let’s put out a fifth edition” and I said “let’s not.” [LAUGHTER] I was not in the mood to do it. I’ve been retired six and a half years now and I’m loving it. I mean, I’m really loving it. And while retired, I was still writing the second edition of Online Teaching at its Best. And then I was writing a book, Infusing Critical Thinking Into Your Course, and I guess I had had it. I mean, I wanted to really make a change and I wanted to get specifically into working at an animal shelter. So I was all occupied with that. So I thought I remember Wilbert J. McKeachie, when he was doing Teaching Tips that he came to a certain point after I don’t know how many editions that he brought other people on to really do the revision work. And so I decided I’m going to do that. So Jossey-Bass said “Okay, fine.” They wanted three names. Okay, I gave him three names, but my first choice was Todd Zakrajsek, because 1. I knew he’d finish it. [LAUGHTER] I knew he’d finish it fast. I knew he do a great job. He knows the literature like the back of his hand, I wouldn’t have a worry in the world. And guess what? Todd accepted. Hip hip hurray. I was so happy. I couldn’t tell you.

Todd: Well, this is great because I said no when they asked me. [LAUGHTER]

Rebecca: Like any smart person would, right? [LAUGHTER]

Todd: Well, I did end up doing it, of course. But the reason I said no was I knew that book very well and I know Linda very well. And I said, “There is no way. I don’t know anybody who can step in and pick this thing up. She knows so much about so much that it’s just not possible.” And they said, “But she really wants you to do this.” So I went back and forth a couple times and I finally decided to do it. And I will tell you, Linda, because I haven’t mentioned this to you. The first three chapters, I had to go back and redo those when I got done with it, because I was so scared of the first three chapters [LAUGHTER] that it was really rough. And then finally it’s like, okay, I hit my rhythm and I walked into it with impostor syndrome a little bit, and I finally caught my footing, but it’s a good book to start with.

Linda: Thank you. Thank you very much. [LAUGHTER] Yeah, I know, the plot thickens, right? It becomes more interesting as you go from chapter to chapter, right. And before you know it, there’s a happy ending after all.

Rebecca: So Linda, Teaching at its Best has been around for a long time with a first edition published in 1998. Can you talk a little bit about how that first edition came about?

Linda: Yes, that was…I can’t believe… 1998. That’s 25 years ago. It’s almost scary how time flies. But anyway, the actual seed of the book came about in about 1994… 95. But I need to give you some background because I had been writing TA training books since like, the late 1970s when I was first given the task of putting together a TA training program. So back then, I was putting out weekly mimeos,[LAUGHTER] remember mimeograph machines. Some of you don’t know, what is she talking about? But anyway, that was technology then. But anyway, smetl great, though… it really did. [LAUGHTER]

Rebecca: That’s the second time today someone has made a reference about the smell of those.

Linda: Yeah, oh yeah.

John: The dittos are what I remember having the stronger smell

Todd: The ditto did, yeah. yeah, and I’ll tell you before we move on, when I was a graduate student, we had a ditto machine. I just have to say this, Linda, because you liked the smell and all there.

Linda: Yeah, Yeah.

Todd: But they had a ditto machine. And below the ditto machine, I noticed that the floor tiles were kind of eaten away by the ditto fluid. [LAUGHTER] And then here’s the best part is that one day I was rooting around in the closet looking for something and I found the extra tiles in a box and the side of the box said “reinforced with long-lasting asbestos.” [LAUGHTER] So the ditto fluid was eating through asbestos lined tile, but that’s how strong that stuff is. So yeah, we all enjoyed the smell of that stuff back in the day..

Linda: Yeah, yeah. I guess it’s a good thing for all of us they invented something else, like copying machines. So anyway, so I started doing that at UCLA. And then that turned into like a booklet of sorts. And then I was at UC Riverside, and I was writing books there. And I sort of revised it every couple of years. And I was also writing these with my master teaching fellows. So we were doing that. And then I came to Vanderbilt, and I decided, well, I’m going to do this, pretty much on my own, I’ll get some help from my master teaching fellows. But anyway, it turned into an actual book. I mean, it turned into a happy monster. And I was very pleased with it. Well, along about 94-95, my husband recommended that I turn it into a regular book, and talk to a publisher about it. So anyway, I said, “Oh, great idea. Great idea and just sort of didn’t think about it much. Then in 1996, he died. And I thought, “Well, how am I going to pull myself through?” I bet it would be a great idea and a great tribute to him if I took Teaching at its Best, the Vanderbilt edition, and turned that into a general book. And I decided to do that and kept my mind off of bad things. And it turned into Teaching at its Best, the first edition. That’s why I dedicated the book to him, by the way, because it really was his inspiration that got me to do it. And so anyway, tribute to him. So that’s where the first edition came from. I mean, it really grew out of tragedy. But it’s been a comedy ever since, right? [LAUGHTER] So anyway, it’s been a wonderful thing.

John: And it’s been a great resource.

Rebecca: It’s interesting that it pulled you through, but then has pulled many teachers through. [LAUGHTER]

Linda: And I’ve gotten such feedback from faculty members who said, “I saved their lunch,” you know, if they were really in big trouble, and some of them said, “I was in big trouble with my teaching and you got me tenure.” Yeah, like, right. But anyway, the book helped a lot of people. And I guess maybe something in me when I first published this book said, “Gee it would really be great to be the next Wilbert McKeachie, right, which is a very pretentious thing to think. But then they wanted the second edition, I was thinking, “Hey, maybe I’m on the road to something.” And then there was a third, and then there was the fourth. And it didn’t get any easier to write the subsequent editions really, it was just a matter of keeping up with the literature. And so right now, I’m off into another corner of the world. So I just didn’t want to immerse myself in that again.

John: So that brings us to the question of what is new in the fifth edition?

Todd: Well, that’s my question. I’ve known Linda for the longest time. By the way, I do want to mention before we go on, I can’t remember, Linda, if it’s been that long ago, but it might have been the second edition. When at POD, I said, “You need to do a second edition of this book” …or second or third. But I was using the book. I mean, I learned so much from it. So for the new edition. Number one, of course, the research has been updated only because the research is always changing. And it had been a few years. So that’s number one. In terms of changing the book, though, we only have a leeway of about 10,000 words. Now, for those out there listening 10,000 words sounds like a lot of words until you’ve got a 200,000 word book, it was about 190. And they said, you can’t go over 200 Because the book just gets too big then. So it is 10,000 words longer than it was in fact, I think it’s 10,003 words longer. So it’s right in there. [LAUGHTER]

Rebecca: So you snuck an extra 3 words in.

Todd: It could have been a squeeze to put three words in there. And it’s always hilarious because when they say there’s just a few too many words I just start hyphenating things so yeah, it kind of all works. [LAUGHTER] Yeah, just any words at all. So you can do “can you” as just a hyphenated word. It works. [LAUGHTER]K So is that terminology, the terminology does change and I find this fascinating. One of the things I love to write about books is learning. I mean, Linda, the same thing what as we write, we read a ton of stuff. And as we read stuff, we learn stuff. So this one in particular, for example, is that I grew up with PBL as problem based learning. And I had done workshops on it, I had worked on everything else, but I hadn’t looked at it for quite a while. And in this particular book, as I started looking at PBL, I couldn’t find anything on problem based learning. And it was fascinating because I was doing some digging, and then I called Claire Major, who was an early person who had a grant on problem based learning and everything I ran into was about 2002, it just started to drop off a little bit, and there was some, but it started to tail off. And when I talked to Claire, she says, “Oh, yeah, I used to do quite a bit about that, it was back around 2002-2003.” And now, and the reason I’m saying this is, every time I saw the letters PBL, it was project based learning. And project based learning sounds a lot like problem based learning, but they’re different concepts. And so anyway, going through and finding some of the terminology, so it was consistent with what’s being done right now has changed. There is now a chapter on inclusive teaching, because over the last three or four years, we finally realized that there’s a whole lot of individuals who haven’t been successful in higher education, partly because of the way we teach. And so I’ve been making an argument for a few years now that teaching and learning, the classroom situation has always really been based for fast-talking, risk-taking extroverts. And we’ve suddenly realized that if you’re not a fast-talking, risk-taking extrovert, you may not get a chance to participate, classroom and other things. So I looked at some different things with inclusive teaching. There’s a whole another chapter on that. And then just the language throughout, we talk a little differently now, just even over the last three or four years than we did five, six years ago, I was pretty surprised by that. But there’s some pretty significant changes in language. So the book has a slightly different tone in language, and those are the biggest changes. Oh, I should say, before we move on, one of the biggest other changes, and I did this one, Linda put a section in there that said learning styles had changed significantly from the previous edition. And so she had pointed out that there was no longer a section on learning styles. And I put the learning styles right back in there, I told Linda and she gasped just a little bit. And then I explained that I put it back in there, and then said exactly how terrible it was to basically teach according to learning styles, because it’s the myth that will not die. So that’s back in there.

Linda: People love it. I know. [LAUGHTER]

John: We have that issue all the time, students come in believing in them and say, “Well, I can’t learn from reading because I’m a visual learner.” And I say “Well, fortunately, you use your eyes to read,” and then I’ll get them some citations.

Todd: Well, I’ll tell you, and before we move on, these are the types of things we learned. I couldn’t figure out why the thing is so hard to die. What is it that’s really doing this because other myths we’ve been able to debunk. And part of the reason is licensing exams, when you are in pre-service and you want to become a teacher, the exams you take to become a teacher, a large portion of those exams, have learning styles questions on there. So you have to answer about visual learners and auditory learners and kinesthetic. And so until we get those out of teacher education programs, we’re teaching teachers to believe this. So anyway, there you go. Public service announcement. Be careful about meshing. And if you don’t know what meshing is, look it up and then stop it. [LAUGHTER]

John: We have had guests on the podcast who mentioned learning styles, and then we edit them out and explain to them later why we edit out any reference to that. And I think most of them were in education, either as instructors, or they’ve been working as secondary teachers. It is a pretty pervasive myth. In fact, Michelle Miller and Kristen Betts, together with some other people, did a survey. And that was the most commonly believed myth about teaching and learning. It was done through OLC a few years back, about three or four years ago. Yep,

Todd: Yeah, I saw that survey. Yes, it’s pretty amazing. Michelle’s an amazing person.

Rebecca: The experience of the pandemic has had a fairly large impact on how our classes are taught. Can you talk a little bit, Todd, about how this is reflected in this new edition?

Todd: Things have changed pretty significantly because of the pandemic. There’s a couple things going on. Again, the inclusive teaching and learning, which I’ve already commented on, is really different now. And it’s interesting, because it goes back to the 1960s. We’ve known that, for instance, African Americans tend to flunk at twice the rate of Caucasians, in large machine-scored multiple choice exams. So we know it’s not the teaching, and we know it’s not the grades, it has to be something else. And it turns out that it was you put students into groups and those differences start to disappear. So I mean, even more so the last couple of years, it’s a lot of engaged learning, active learning. I’m still going to pitch my stuff that I’ve been ranting and raving about for years. And there’s no data out there that says that lecturing is bad. What the data says is that if you add active and engaged learning to lecture, then you have much better outcomes than lecture alone. But we’re learning about those types of things in terms of active and engaged learning, how to pair it with and mix it with other strategies that work, looking at distance education in terms of systems and how we can use technology. So a quick example is I used to have a review session before exams. And oftentimes, it’s hard to find a place on campus to have that. And so you might be in a room off in one hall or the library or something. And if the exam was on Monday, I’d have the review session at like six o’clock, seven o’clock on a Sunday night. And there are students who couldn’t make it. I would simply say, you can get notes from someone else. And we’ve known for the longest time, if a student misses class, getting notes from somebody else doesn’t work. Well, now I do review sessions on Zoom, we don’t have to worry about finding a place to park, we don’t have to worry about some students finding babysitters, if they’re working, it’s recorded, so they get the exact same thing. So things like Zoom have really changed teaching in a sense that you can capture the essence in the experience of teaching and use it for others, and it has helped with some equity issues. You can’t do it all the time. And teaching over Zoom is different than face to face. But there are now ways of using different technologies and using different modalities to help to teach in ways that were not really used before the pandemic.

John: Speaking of that, during the pandemic, there was a period of rapid expansion in both the variety of edtech tools available and in terms of teaching modalities themselves. In the description of your book, it indicates that you address useful educational technology and what is a waste of time? Could you give us an example of both some useful technologies that could be used and some that are not so useful? And also perhaps a reaction to the spread of bichronous and HyFlex instruction?

Linda: Yeah, I’ll take this one. And I’m drawing a lot of stuff from another book that I co-authored, with Ludwika Goodman. We were writing about Online Teaching at its Best, okay. And she was an instructional designer. And I came from teaching and learning and we put our literature’s together. And we were talking about modalities a great deal, especially in the second edition with the pandemic. Well, one thing I found out, not only from reading, but also from watching this happen was that this Hyflex or bichronous, whatever you want to call it, is a bust, if there ever was a modality, that’s a bad idea it’s that one, even though administrators love it because students can choose whether to come to class and do the things they would do in class, or to attend class remotely? Well, yeah, it sounds like “oh, yeah, that could be good.” But the technological problems, and then the social problems, especially the in-class social problems are enormous. And in-class social problems, like small group work, how do you hear what’s going on in the classroom over this low roar of small groups? Okay, so how can you help? How can the students that are learning remotely, what can they do? Now, the way this was invented, by the way, was for a small graduate class, and then okay, like, makes sense, because you’re only dealing with six students in this room and six students who are remote. But other than that, it’s so bad, the logistics, the sound logistics, the coordination that the instructor has to maintain, the attempt at being fair to both groups, at bringing in both groups, when the groups can’t even hear each other well. Now, if we had Hollywood level equipment in our classrooms, we might be able to make this work a little better, but we don’t, and we’re never going to have that. So there are just a lot of technological and social reasons why HyFlex, that’s what I called it in Online Teaching at its Best, what it was called at the time is a complete bust. Now, not to be confused with hybrid or blended learning, which we found has worked exceedingly well. So bringing in some technology, but into a face-to-face environment and that being the base of the class. Now, remote’s nice, but you might not want to do remote all the time for all things. It’s not quite the next best thing to being there. But it’s something and as long as you don’t just stand there and stare at the camera and lecture for an hour. You’ll get complaints about that quickly. And particularly with students today when they really need to be actively involved, actively engaged. So yeah, sure, fine, talk for three minutes, maybe even push it for five, but then give them something to do and you really, really must in remote because otherwise, you’re just some talking head on television.

Todd: I agree completely. In fact, it was funny because I happen to have a digital copy of the book here. And so I typed in a ctrl F and I typed in HyFlex and there’s one comment to the preface that said there’s many different formats out there and then I will tell the listeners, if you’re expecting to learn about HyFlex, the word never shows up again in the book. [LAUGHTER] So, it’s not in there. I mean, you look at the literature that’s out there. And I think it’s fair to say that maybe there are people who can do it. I haven’t really seen it done well and I think Linda’s saying she hasn’t either. And it’s so difficult, especially for a book like this. That’s not what we’re all about. I mean, again, if it even works well, which I’d love to hear that it would be a very advanced book and that’s not what this is. So we do have a lot in there about technology in terms of edtech tools, though. There are those in there, I would just say real quickly, for instance, Padlet’s one of my favorites, I’ll throw that out there. I like Padlet a lot. But there are tools out there, if you want to do a gallery walk, which for instance, if you happen to be in a face-to-face course, you’d set up maybe four stations with big sheets of paper, you put your students into groups, and then they walk from sheet of paper to sheet of paper, and they move around the room. And they can do what’s called a gallery walk. You can do the same thing online with a jamboard, you can set up jamboards so that there’s different pages, and then each group is on a page. And then you just say it’s time to shift pages, they could shift pages. So I’ve done gallery walks, and it’s worked well. I’ve used Padlet for brainstorming. And one of the things I love about Padlet, I’ll have to say is if you are doing some digital teaching in a situation, you can watch what each group is developing on the page for all groups at the same time. I can’t hear all groups at the same time when I walk around the room. So there are certainly some technologies coming out that can really do things well. There’s also things that don’t work very well, though. And I think one of the things you want to keep in mind is just learning theory. Does the technology you’re using advance students, potentially, through learning theory? Does it help with repetition? Does it help with attention? Linda was just mentioning attention, if you lecture too long, you lose their attention. If you do something ridiculously simple or not… I was gonna say stupid, but that sounds rude. But if we do something as a small group that makes no sense, you don’t get their attention either. So using clickers, I have to say, I watched a faculty member one time because they were touted as a person who was very engaging. And this is at a medical school, so I really wanted to see this. And the person used clickers, but used it in a way that asked the students a question, they responded, and the instructor looked up at the board and said, “Here’s how you responded, let’s move on.” And then moved on to the next thing. And about five minutes later gave another question said, “How do you respond?” and they clicked the clicker, and then they moved on again. That had no value at all, and in fact, there was no actual interaction there. So afterwards, I say, can’t you just ask a rhetorical question and just move on? We got to be careful not to use technology just because it’s being used, it should advance the learning process.

John: However, clickers can be effective if it’s combined with peer discussion and some feedback and some just-in-time teaching. If it’s just used to get responses that are ignored, it really doesn’t align with any evidence-based practice or anything we know about teaching and learning. But those per discussions can be useful and there’s a lot of research that show that does result in longer-term knowledge retention when it’s used correctly, but often it’s not.

Todd: Right. And I think that’s a really good point. I’m glad you said that, because Eric Mazur, and his concept tests, for a large extent, that’s where active and engaged learning really took off. And that is a clicker questions. And they can be used as great tools. But again, if you’re using it for the right reason, which is what you just said, My comment is, there’s technology out there, that is a waste of time, and not a good thing to have, because it’s just not being used in a way that’s conducive to learning. So good point, that’s fair.

Rebecca: Can you talk a little bit about who the audience of the book is?

Linda: Sure. It’s actually for anybody who teaches students older than children, I suppose, because it isn’t really designed for teaching children. But other than that, it’s really for people who teach but don’t have the time to read a book. The nice thing about Teaching at its Best is you can go to the table of contents, you go to the index, you could find exactly what you need for your next class. And it’s very oriented towards how to, so it could be for beginners or for experienced people who simply haven’t tried something specific before, or want a twist on it, or just want some inspiration. Because there there are a lot of different teaching techniques in there. And they’re all oriented towards student engagement, every single one of them. But I wanted to comment too, on just how the job of instructor or professor has changed over the past, I don’t know, 40 years, I suppose. I know when I started teaching it was a completely different job. And I started teaching in 1975, when I was 12, of course, but no and I was young to start teaching because I was 25 and there I was 180 students in front of me. So oops, my goodness, what have I done? But that’s exactly what I wanted to do. But you’d go in there, you’d lecture and you’d walk out. You were in complete control of everything. Like, you might throw out a question and you might get a discussion going. But it wasn’t considered to be essential. In fact, there were two teaching techniques back then: there was lecture, and there was discussion. And nobody knew how to do discussion. Now, I had to find out a few things about it when I was doing TA training, because TAs were supposed to be running discussions. But there wasn’t a lot out there. Thank God for Wilbert McKeachie’s book Teaching Tips, because that was about the only source out there you could go to. So anyway, but now the job, I mean, oh, it’s mind boggling what faculty are now expected to do. And they are supposed to, like, learning outcomes. Okay, I love learning outcomes. They’re wonderful. But I didn’t have to do that when I started, I just had to talk about my subject, which I dearly loved. And so, that was nothing. But you’ve got learning outcomes. So you’ve got to be like, a course designer, you have to deal with a student’s mental health problems, right? It’s part of the job, and you’re expected to respond to them. You’re supposed to give them career counseling in careers that you might not know much about, and possibly for good reason, because you’re in your own career. It’s so time consuming, not to mention fair use, oh, yes, fair use has changed, fair use has changed radically. And when you’re dealing with anything online, the rules are totally different. And you’re highly restricted as to what you can use, what you could do. When you’re in a face-to-face classroom, it’s a little bit easier. So yeah, so you got to be a copyright lawyer to stay out of trouble. And then you get involved in accreditation, you get involved in that kind of assessment. So you have to all of a sudden be totally involved in what your program is doing, what your major is doing, where it’s headed. There’s just too much to do. And there are more and more committees and oh, there’s a lot of time wasted in committees. Of course, you’re supposed to publish at the same time and make presentations at conferences. It was like that back then, too. But now, the expectations are higher, and it’s on top of more time in teaching, and more courses. I was teaching four courses a year, and you can’t find that kind of job anymore.

Rebecca: So Linda, you’re saying the animal shelter is going really well now?

Linda: Yes. Yes. Yes. Yes.

Todd: That’s hilarious. Well, I want to point out too, and I think Linda’s said it very, very well is that we are expected to do things we never had to use before. Never worried about before. And I love the fair use is great, because when I first started teaching, and I’ve been teaching for 36 years, when I first started teaching, you’d videotape something off TV and show it in class and then put it on the shelf. And I knew people who showed the same video for 10 years. Right now you better be careful about showing the same video for 10 years. But these are things we need to know. I would say also, by the way, this is a really good book for administrators, anybody who would like to give guidance to faculty members, or better understand teaching and learning so that when promotion and tenure comes along, you get a sense of this. And so if you’re saying to the faculty, they should use a variety of teaching strategies. It’s not a bad idea to know a variety of teaching strategies. And so I think it’s good for administrators as well, and graduate students. But I want to take a second and tell you, one of the reviews of the book, I guess, came in just yesterday or the day before from Dan Levy. He’s a senior lecturer at Harvard University. And what he put was Teaching at it’s Best is an absolute gem. Whether you are new to teaching in higher education, or have been doing it for a while, you will find this book’s evidence-based advice on a wide range of teaching issues to be very helpful. The style is engaging and the breadth is impressive. If you want to teach at your best you should read Teaching at its Best. And I love what he put in there because it doesn’t matter if you’re a new teacher or you’ve been doing it for a while, this book’s got a lot of stuff in it.

John: And Dan has been a guest on our podcast, and he’s also an economist, which is another thing in his favor. [LAUGHTER]

Todd: That is good.

John: I do want to comment on lenders observation about how teaching has changed because I came in at a very similar type of experience. I was told by the chair of the department not to waste a lot of time on teaching and to focus primarily on research because that’s what’s most important, and that’s the only thing that’s really ultimately valued here or elsewhere in the job market. But then what happened is a few people started reading the literature on how we learn And then they started writing these books about it. [LAUGHTER] And these books encouraged us to do things like retrieval practice and low-stakes tests, and to provide lots of feedback to students. So those people…[LAUGHTER]

Rebecca: I don’t know any of them.

John: …but as a result of that many people started changing the way they teach in response to this. So some of it is you brought this on to all of us by sharing… [LAUGHTER]

Linda: I apologize.

Todd: Sorry about that.

Linda: I apologize.

Todd: We apologize and you know, I will say too is, so yeah, sorry. Sorry about doing that. But I’m glad you said that.

Linda: We made the job harder didn’t we?

Todd: We did, but you know to just be fair for Linda and I as well as I still remember a faculty member calling me, It must have been about 20 years ago, and I just started doing a little bit of Faculty Development, she was crying, she had given her first assignment in terms of a paper. And she said, I’m sitting here with a stack of papers, and I don’t know how to grade them. And it got me thinking a little bit, how many of the aspects of the job that we’re required to do, were we trained to do? And that’s the stuff that Linda was mentioning as well, is nobody taught me. I’m an industrial psychologist. And so nobody taught me the strategies for delivering information to a group of 200 people. Nobody taught me how to grade essay tests. Nobody taught me how to grade presentations, I didn’t know about fair use and how I could use things. I mean, you go through and list all of the things that you’re required to do. And then look at all the things you were trained to do. And this is tough. And that changed. So I have one quick one I’ll mention is I was hired as an adjunct faculty member before I got my first tenure-track job. And I was teaching 4-4. So I had four classes in the fall, four classes in the spring. And about halfway through the spring, I ran into the department chair, and I was interested to see if I was going to be able to come back and I said, “Hey, Mike, how am I doing?” And this was at Central Michigan University, a pretty good sized school. He said, “You were fantastic.” And I said, “Excellent. What have you heard?” He said, “absolutely nothing.” So when it comes to teaching, what I learned was: research, you had to do well, and teaching, you had to not do terribly. And that is what you were mentioning has changed is now you’re kind of expected to do teaching as well.

Rebecca: And there’s a lot more research in the area now too. So sometimes it’s hard to keep up on it. So books like this can be really helpful in providing a lot of that research in one place.

John: And both of you have written many good books that have guided many, many faculty in their careers, and eliminated that gap between what we’re trained to do and what we actually have to do.

Rebecca: So of course, we want to know when we can have this book in our hands.

Todd: Good news for this book, which is exciting because we really cranked away on this thing and it’s listed in Amazon as being due on April 25. But it actually went to press on January 23. So it’s already out and about three months ahead of schedule.

John: Excellent. We’re looking forward to it. I’ve had my copy on preorder since I saw a tweet about this. I think it was your tweet, Todd, a while back. And I’m very much looking forward to receiving a copy of it.

Todd: Excellent. We’re looking forward to people being able to benefit from copies of it.

Rebecca: So we always wrap up by asking what’s next.

Todd: It’s hard to tell what’s next because I’m exhausted from what’s been [LAUGHTER] ever moving forward, as I’m working on and just finishing a book right now that’s to help faculty in the first year of their teaching. So it’s basically off to a good start. It’s what specifically faculty should do in the first year of getting a teaching position. And aside from that, probably working on my next jigsaw puzzle, I like to do the great big jigsaw puzzles. And so I just finished one that had 33,600 pieces. It is five feet….

Rebecca: Did you say 33,000 pieces?

Todd: No, I said 33,600 pieces.It was the 600 that…

Rebecca: Oh, ok.

Todd: …was difficult. [LAUGHTER]

Rebecca: Yeah.

Todd: When the puzzle is done, it has standard sized pieces, and it is five feet by 20 feet. So I just enjoy massively putting something together. It’s very challenging. So quite frankly, for those about and listening to this is if you imagine 33,600 puzzle pieces, that’s about as many studies as Linda and I have read to put this book together. [LAUGHTER]

Linda: Nothing to it. [LAUGHTER]

Todd: So that’s it for me. [LAUGHTER] Linda, what are you up to these days?

Linda: Oh, well, I live in la la land. So I’m still doing workshops and webinars and things like that mostly on my books of various kinds, various teaching topics. But I think what I want to do is retake up pastels and charcoals. My father was a commercial artist. And so he got me into pastels and charcoals when I was in high school. Well and then I dropped it to go off to college. Well, I want to get back into it in addition to working at the animal shelter. I know. It’s la la land and I wish la la land on everybody that I like.[LAUGHTER] I hope you all go to la la land and enjoy being a four year old all over again, because that’s the way I feel. I adapted to retirement in about 24 hours. That’s pushing it… you know, it’s more like four. But anyway, I slept on it. [LAUGHTER] That was the end of it. But I know I eased into it. I eased into it. I was still writing. I was still doing, especially before the pandemic, a lot of speaking. So then the pandemic hit and it just turned into online everything. And now I’m back on the road again, to a certain extent. I love it. So anyway, it’s a nice balance. So yeah, I wish you all la la land too.

Todd: That’s great.

Rebecca: That’s something to aspire to.

Todd: Yeah, it is. But you know, since you mentioned the speaking things, I just have to do the quick plug here. Linda, I think you and I, years and years ago, were joking around at POD about who would be the first one to get to the 50 states and have done a presentation in every state. And so I gotta tell you, I’m not even sure where you’re at in the mix, but I am at 49 states. And if any of your listeners are in North Dakota, [LAUGHTER] I could certainly use a phone call from North Dakota.

Linda: Well, I want to go to Vermont. I have not been to Vermont…

Todd: Oh, you haven’t.

Linda: …to give a presentation. So I would enjoy that. But I’ll go to Hawaii. I’ll do anything in Hawaii for you. Absolutely anything. [LAUGHTER] I’ll do gardening, [LAUGHTER] I’ll do dishes, your laundry. I don’t care.

Todd: That is good. Yeah, Linda and I had this gig. It was a long, long time ago. And I don’t know, it must have been 20 years ago we talked about it even. And there was some rules too. You had to be invited. And there had to be some kind of an honorarium or just I mean, it didn’t have to be much, but the concept was you just couldn’t show up at a state and start talking. [LAUGHTER] Otherwise, we’d have both been done a long time ago. But yeah,

Linda: Yeah.

Todd: … it was fun. This is the way nerds have fun. [LAUGHTER]

John: Well, that’s a competition that’s benefited again, a lot of people over the years.

Rebecca: Well, thanks so much for joining us. It’s great to see both of you again, and we look forward to seeing your new book.

Linda: Thank you for this opportunity. It was a pleasure.

Todd: It was so much fun. Thank you

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

189. Teaching with Zoom

Posted in backward design, classroom community, clickers, edtech, higher education, instructional communication, learning, online discussions, pedagogy, teaching
May 26, 2021

The COVID-19 pandemic resulted in an explosion in the use of remote synchronous instruction, a modality that was rarely used until March 2020. In this episode, Dan Levy joins us to discuss the affordances and the challenges associated with this relatively new modality. Dan is an economist and a senior lecturer in Public Policy at Harvard University where he teaches courses in quantitative methods, policy analysis, and program evaluation. He is the author of Teaching Effectively with Zoom, A Practical Guide to Engage Your Students and Help Them Learn, which is now in its second edition.

Shownotes

Transcript

John: The COVID-19 pandemic resulted in an explosion in the use of remote synchronous instruction, a modality that was rarely used until March 2020. In this episode, we discuss the affordances and the challenges associated with this relatively new modality.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Our guest today is Dan Levy. Dan is an economist and a senior lecturer in Public Policy at Harvard University where he teaches courses in quantitative methods, policy analysis, and program evaluation. He is the author of Teaching Effectively with Zoom, A Practical Guide to Engage Your Students and Help Them Learn, which is now in its second edition. Welcome, Dan.

Dan: Thank you very much, Rebecca and John.

John: We’re looking forward to talking to you.

Dan: Thank you.

John: Our teas today are… are you drinking tea?

Dan: I love Moroccan tea. My family’s originally from Morocco. And that’s the tea that I normally drink when I drink tea.

Rebecca: Today, I have Scottish afternoon, John… we’re coming back, coming back with the good stuff.

John: And I have two teas here, actually. I have ginger peach green tea and a Moroccan mint tea.

Dan: Oh, wow.

John: That worked nicely.

Dan: Yeah.

Rebecca: I don’t know if I’d drink them quite at the same time, but… [LAUGHTER]

John: Well, they’re sequential.

Rebecca: So we’ve invited you here today to discuss Teaching Effectively with Zoom. Could you talk a little bit about how you started this book project?

Dan: Sure. So in March of last year, when we all had to go quickly to remote teaching, I had spent the better part of 10 years trying things surrounding online learning at the Harvard Kennedy School. But I had never spent much time with synchronous online teaching. And so when we had to move to remote teaching, my first instinct was to go and observe as many instructors as possible to see what they were doing. And what I discovered then was an incredible wealth of people who were just doing incredible things, they’re being very resourceful in the way that they were trying to use the platform to accomplish our pedagogical goals. I didn’t set out to write a book at that time, but I was just learning a lot. And at the same time, I was observing my daughters, in high school, receiving online learning instruction. Around mid-May, I sort of had the feeling that in the fall, we would be teaching online still. And I felt that there was a lot being written online, in Twitter and blogs and all of that. But I felt, gosh, this is overwhelming. And so I felt the need to have in one single place, what I thought would be useful gui e for instructors who were saying, “I need to do this, I want to do it well.” And I thought that I had gotten a lot of ideas from the colleagues that I observed teach… which by the way, observing colleagues teach is one of the silver linings of the pandemic, because it’s now easier than ever, and it’s an incredibly powerful way of learning. So in any case, at that point, I said, “I want to write this book, I’ve never done anything like that. And I want the book to be ready by July 1, because that’s how it’s gonna be helpful to people in the world, given the academic calendars.” And on July 1, a book was ready. And then on July 2, we put it out there to the world. And then because so much happened in the fall, I released the second edition based on everything that I had learned since then, from my own teaching and that of colleagues. And it was very rewarding to see people from all over the world who had engaged with the book, also contribute with some of their examples.

Rebecca: I think I need to get one of those magic wands you must have to turn around stuff that quickly.

Dan: Well, no. Thank you. I think there’s nothing like a deadline… [LAUGHTER]

Rebecca: True

Dan: …and a deadline that I really felt it was important to meet. And I have a good friend who I sent the book, he is not in the education world. And I said, “Here it is.” And then he told me, “Then you lied to me. You told me that you wrote the book in a month and a half. But I know from previous conversations with you that you have been writing this book in your mind for the last 10 years.” And I thought that was an interesting way of putting it because I’ve been thinking about a lot of these issues, but I never sat down to write any of them.

John: It was extremely timely, and I know many people adopted the book last summer or picked up the second edition in January when that came out. There weren’t a lot of resources other than lots of Twitter posts and lots of blog posts on specific aspects. But the book blends together a nice discussion of the technical details of how you do things with effective pedagogy, which is a resource that was very much needed and is still very much needed for many people as we move forward. Because this is an area that people had not really done very much with until this sudden transition.

Dan: Yes, thank you so much John.

John: What are some of the most effective ways that you’ve seen faculty using Zoom in their classes or that you’ve used Zoom in your classes yourself?

Dan: One observation that I had throughout this year… and the book is a little bit organized in this way, but it didn’t crystallize to me until later in the last year… which is that if you conceptualize the way students can engage in your course and think about the different channels… in the book, I describe five main channels, they can speak, they can vote, they can write, they can work in groups, and they can show their work. One of the things that becomes very obvious, at least to me, is that for default, in in-person teaching, tends to be verbal, we speak to each other. And what I realize in live online learning is that of those five channels, the one that most degrades when you go from in person to online is precisely the verbal one. And I think, my sense is that of recognition that that’s the case for many, many reasons, is what I think has made some instructors particularly successful at doing this because they are not wedded to verbal as the main or default channel of communication. So that’s kind of like an overall message that if you think about in which ways can your students engage in your class, and in which of these ways do I want for this particular pedagogic purpose my students to engage with, my sense is that that tends to be a winning combination.

Rebecca: When I was looking at that organization of your book, Dan, it really struck me and was really helpful way of thinking about it. And, in your description right now, made it really clear to me why it was actually very easy for me to switch to synchronous online learning [LAUGHTER] because I don’t really prefer the verbal. [LAUGHTER] So it was nice to engage in these other spaces as an introvert, like I could use chat in other places that I’m actually much more comfortable. [LAUGHTER]

Dan: It’s interesting you say this, Rebecca, because the same thing that you said, is true for students. So introverted students now have different ways of engaging with us that we might not have even heard from them before. And I think if we leverage those ways, we’re going to end up being in a better place. And most importantly, they’re going to end up being in a better place.

John: One of the nice things about written communication and chat is you’ve got that delete key, which, when people are feeling a little more introverted, perhaps, they’re less confident about saying something where they can’t take something back, rephrase it on the fly. And having that delete option, lets them be a little more thoughtful in their participation, and can lead to a much more inclusive environment in many ways.

Dan: Absolutely. The other thing that it does is that you can take your time to compose a message that you write, whereas, when you’re called to speak, you might have perhaps practiced this message in your mind, but you feel like on the spot, you have to now deliver it at that point. And then the introverts tend to have more difficulty with that. And I say that as an introvert. I don’t want to be too binary in the definition. But I say that as an introvert.

Rebecca: Can you talk a little bit about why the verbal channel degrades a bit in a Zoom situation? Because I think that might actually be really helpful for people to think about.

Dan: Sure. So one way in which it does, and I wish the podcast was a video podcast, but one way in which it does is right now we are with this software, and the three of us can see each other. But right now, I’m looking at you, Rebecca, and you think I’m looking down somewhere and not at you. Like you have no idea that I’m looking at you. And you’re like, “Why is this person looking down? I’m speaking with him.” And John right now also thinks I’m looking down. So he doesn’t even know that I’m looking at you and not at him. And if I wanted to give you, Rebecca, the impression that I’m looking at you, I would have to point my eyes to a camera, and I no longer have any nonverbal feedback from you, I have no idea of what’s going on with you. And not only that, now John thinks I’m also looking at him, and I’m not looking at either of you. [LAUGHTER] So, that’s kind of one simple level and I’m optimistic that maybe we’ll have technology that solves this. The other day, I saw a Kickstarter campaign that a friend of mine forwarded to me for this idea that I kind of have been having for a while but it seems like someone actually created a product for a video camera that is in the middle of your screen, rather than at the top or at the bottom. So, in any case, that’s one aspect of it, but another, I think, important aspect… and people have written about it… is that the communication is just not as effective. You cannot signal in the same way non-verbally as you can signal in a classroom. In a classroom, you have your whole body to express, you can use physical distance with the students, to approach, you can move, there are so many other things at your disposal. And the one to me that still becomes the most important one is that you cannot hear the classroom. People have to unmute… if you have a big enough classroom they have to unmute, and that is just much less natural. There’s no “click this” reaction item to sort of say how you’re feeling. No, you just see it automatically. So in any case, those are some of the ones that I have felt myself, I’m sure that you as educators have also other ones. But of all the channels it’s the clearest one in which in-person seems to me better than online.

John: Are there any other ways in which remote synchronous instruction offers some advantages that we don’t have in the classroom?

Dan: Yeah, for example, writing… we were talking about writing. We can use writing in the classroom. I think many of us have shied away, we’re nervous about having our students with their laptops in the classroom and so on. But writing feels to me such a powerful tool, not only for doing the kinds of things that Rebecca was saying before, that you can bring introverts, or the things that you were saying more generally, John, that you can have more inclusive teaching, but you can do what some people might describe parallel processing instead of serial processing. So if you ask students in the classroom, can you give me an example of X, please write it in the chat, within 30 seconds you have 20 examples if you have 20 students, whereas if you had to do it verbally, you would take one at a time. And that, I think, is much less efficient in that sense. So I think there are many, many reasons why chat, even though it’s controversial, can be powerful. And I know one of the favorite ones that I’m sure you’ve all used is this one-minute paper, where you tend to distribute this piece of paper where they write it and they give it to you. And I’ve always had the intention of using this in my physical classrooms. But many times, it seems like the last minute of class there’s something more urgent that I need to do and then there are logistics there. But with online live teaching, it’s very easy. You can do something as simple as “One minute left in class, please everyone write down what was your main key takeaway from today.” And within a minute, you have a lot of information of what happened in that class.

Rebecca: So a lot of faculty also seem to be under the impression that by being physically in the same space, somehow community is automatically formed. Can you talk a little bit about how community does build in an online synchronous space?

Dan: To me, this was one of the biggest positive surprises I thought of all the aspects of online teaching, this would be the one where it would perform the worst. And I do think that there’s something special that happens when human beings are together in the same space. There’s no question for me about it. But I observed many instructors doing things that I think helped create community in the classroom in ways that I was very surprised. And if I had one general guidance to give is that you just have to be a lot more deliberate about creating community than you are when you are in the same physical space together. And people do it in all sorts of ways. But I think just being deliberate and being intentional about it goes a long way. And just to name three very practical things. One is, if you can open your classroom before class starts, in some way that simulates what you would do in a regular class anyway. Second, if you can stay in your online class for a few minutes after to speak with students. It’s another way of doing it. And then there are of course, things you can do that in a physical and in an online classroom that I think are good for creating communities. If you can learn about your students, so that they know that you know them, that you’ve taken a personal interest in them and that you can bring that to the classroom, that I think is just as true or nine as it is in person. Then there are many other things, there like music lists, and many, many things that people have been very creative about. But those are three that come to mind as fairly easy to do.

John: One of the things I really like about your book is you start by emphasizing the use of a backwards design approach in classes. And you suggest that that be done at the level of individual class sessions or individual activities. Could you give us an example of how you might apply that in a synchronous online session in Zoom?

Dan: Many, many people that listen to your show, I’m sure have heard of backward design and subscribe to it in their own teaching. I think in some way, it’s not that different online in the sense that you think about what are two or three things that I want to make sure that students are able to do at the end of this learning experience. And when you plan your class, you organize it around those things. And one of my biggest challenges as an instructor is time management, it’s like, “Oh my god, can I manage time to do this?”… but what has been very helpful to me is I might have a class plan that says part 1 – 15 minutes part 2 – 23 minutes, and so on. And as I look at the clock, I know where I am in the class plan relative to the time, and knowing what are those two or three things that you want to make sure that everyone gets at the end, allows you to make choices in the class that I think become more likely to succeed. So for example, if you feel like you’re running behind, and there is a particular topic that you think is useful, but not crucial to those two or three things, you might decide to skip it, or you might decide to go a little bit faster, or you might decide not to pause for the discussion that you were planning to have. So, having a concrete set of what you are trying to achieve. I know it sounds obvious, but it wasn’t to me when I first started teaching. I conceived of teaching like, “Well, we need to cover this, and this is what we’re gonna do.” And I still remember attending a one-hour session when I was a PhD student at Northwestern University, from the Director of Teaching and Learning Center at the time, Ken Bain. And I remember him introducing this idea. And that was totally revolutionary to me. Again, I know for many of us, it’s not anymore. But that was more than 20 years ago, and has guided my teaching ever since.

Rebecca: You know, Dan, I certainly subscribe to backwards design, both as a designer and also as a teacher.

Dan: Yeah.

Rebecca: … but I did find myself doing synchronous online being really specific about time chunks, because it’s like, “We need to mix this up, otherwise, we’re just staring at a screen.” And being even more intentional about that. I’d have an agenda and the students can see me going in there like, “No, we’re changing this agenda. [LAUGHTER] on the fly is like “No, this conversation’s good, we need to do this instead.” [LAUGHTER]

Dan: Yes, I guess they’re seeing your design as you are executing it.John, you were asking about some of the advantages of online. I hate to mention this as an advantage. But the reality is, we now have screens, and we can put to the side of the screen things that we want to remember in a way that’s harder to do in a class. So this is like a super tactical tip, but you’re interested in teaching inclusively in the classroom, and you’re worried about voices that haven’t participated that much, you can do something as low tech as: before your class, you write the name of say, three to five students that you want to call on in a piece of paper. And you tape that piece of paper to the right of your screen, right where the participant, the Zoom participant list normally is, or Zoom or whichever other software. And when you see the participant list and you see a few hands up, you look at your paper list and sort of see is one of the hands up belonging to one of the students that I want to call on. And that seems like a simple thing. But it is helpful. I don’t know if you know, but I co-created this application called Teachly, which allows you to track participation and help people teach more inclusively and effectively. And this makes the use of this app even easier, because once you have that participant list, you just put it next to the participant list in Zoom, either electronically or physically. And you have that as a way to do it.

John: And if I remember correctly, on your companion website, you have a picture of a list of names taped to the side of the screen.

Dan: Yes.

John: And we should also mention that there is a really great list of resources associated with the book. We’ll include a link to that in the show notes.

Dan: Thank you. Yeah, the website. Victoria Barnum, I work with her, she put it together. And it was again, a very quick way to try to put resources out there that would be helpful for people. And there’s one page for each chapter of the book.

John: You mentioned rearranging things to make sure you get to the end result. And that’s something I’ve noticed I have to do a lot more with synchronous remote sessions than I did in the classroom. I think partly because I was so used to doing it in a classroom, I had routines where I could get the things more or less quickly with a whole set of activities. But maybe I’ve been over-preparing, but I have a big list of things I want to do. I do polling in the classroom, I have some group work where they’re working in breakout rooms, and I never can get all them done. So I’m constantly, as I’m going through each day’s session trying, “Well, which of these is most important to getting them to that goal,” which is a way I never really had to think about quite as extensively as I do now.

Dan: Yes, though I will say that I was having those same struggles before online teaching, but I also share your experience that they have become more prevalent, and to the extent that it has forced that conversation on all of us. And “Okay, what is the essence of what I want to make sure that students are able to learn in this class?” I think that’s a positive development. One of the things that I discovered very early on in the process of writing a book is that many instructors, and maybe John, this applies to your experience, were saying that compared to their in-person class, when they try to execute that plan, they generally were only able to do about 80% of what they were doing before. Now I don’t know how much of that is still true today. Maybe as we get better with teaching online, we can get that number closer to the 100%. And I don’t know the extent to which it has to do with Rebecca’s questions about verbal communication degrading and making it harder to communicate. But to the extent that that number is even in the ballpark of being true, it does explain why most of us are feeling that need to interrogate more our class trends.

21:15.461

Rebecca: Maybe it’ll also make us a little more empathetic to students who have time management issues, we’re sure. Zoom has really evolved quite a bit since March, there’s new features and new capabilities and things. Can you talk a little bit about how your own teaching using zoom has evolved over the past year,

Dan: I think as all of us practice has allowed me to become better at it, I remember the first few times, I couldn’t even imagine being able to check the chat at the same time, then I was teaching, I was like, there’s just too much going on here. Now, I won’t say that I can handle any number of comments in the chat. But now I can do it in a way that I couldn’t do it before. And so in some way, teaching online live sessions is an exercise in multitasking, you have to pay attention to a lot of things that are happening in your students in your screen, and so on. And frankly, with as much practice you get better. And I think that’s one thing that’s useful in terms of zoom specific things. I think one feature that has come out relatively recently, which to me opens a whole set of possibilities is the fact that now in breakout rooms, you can set it up so that children’s can choose the breakout rooms. And I think that opens up many interesting possibilities in that perhaps students can choose according to a particular interest that they have, perhaps they can choose relative to position that they might have in a debate relative to a vote they have had in a poll. And that I think, in some way, is incredibly powerful. So that’s one way in which I’ve just began to explore. And I hope by the way, there are other ways of setting up the rooms in the future, that might be good. The second thing, which we haven’t talked too much about the breakout rooms. But I think breakout rooms combined with collaborative documents, such as Google Slides, or jam board or mural, whatever other tools we have, can be incredibly powerful for group work. And that has been an area of constant experimentation for me and many of my colleagues. And that is one area where I think we can make even more progress. And my sense is that we’re going to bring some of that into our physical classrooms, when hopefully someday we come back to our physical classrooms.

Rebecca: I’ve experimented a lot with those new breakout rooms more recently, as well. And even with some mastery learning activities, where we’re doing exercises, and as they complete one, they can move to the next one and moving to different breakout rooms depending on what they’re working on. So they can help each other out and collaborate. And that’s been working really well. And I’ve gotten a lot of good feedback from students about how that’s actually really helpful to like community of people who are actually at the same moment of their learning. So there’s a lot of possibilities there.

Dan: Many, many you can have different themes. There’s one thing we experimented recently, I don’t know if any of you co teach a class, but we were doing this in a program in executive education, we to try to create a more intimate environment, we divided the class into two groups, and each of the group was with one faculty member. And we had asked each of these faculty members to be available for an hour. So for the first half hour, Group A with with the first instructor group, he was with the second instructor, and then at the half hour marked, we swapped them and we thought about Okay, how can we do this easily without people getting lost? And all we did was to swap the instructors from one room to another and all of a sudden you basically had students who are staying with one hour in a breakout room, were able to have a more intimate experience with two instructors. With that, I think would have been hard to do in the physical world.

Rebecca: Nice that we’re getting to the point where we’re appreciating some of the digital rather than scrambling all the time.

Dan: Yes, I think there’s some things that actually worked better.

John: And certainly that ability to mix up groups easily and quickly in different ways. Either having persistent rooms where you have persistent groups working in the same room regularly or mixing it up for different topics or again, doing the self selection gives you a lot more variety and how you mix and match activities.

Dan: It does. My colleague Terry serranos was experimenting with this and I thought that was an interesting use. If you have a teaching assistant and you have your students work on an activity during class, they three or four minutes in silence work on this you can open a couple of breakout rooms and one of them has a teaching assistant and if you would like help from the teaching assistant go to breakout room one if you would like to work alone in a virtual room go to this other room. So I think we’re experimenting in ways that I think are conducive to good learning experiences.

John:
One of the other in recent addition, cism is the ability to let co hosts set up and establish the breakout room. So if you do have a teaching assistant or multiple instructors, if you’re presenting on something, or if you’re working with a group, you don’t have to do the back end arrangement while you’re also trying to do other things. So if that makes it a whole lot easier,

Dan: I totally agree. And sometimes you can multitask. But if you can have one less task to do, that’s probably helpful.

John: You also talk quite a bit about the use of Paul, and could you talk about some of the ways in which people might do Pauling and how polling might be used effectively in instruction,

Dan: I want to first say that I started using polling many, many years ago in my physical classrooms inspired by one of my mentors, Eric, Missouri in the physics department, at our end, I do you have a bias towards using them. But I would say a first approximation, polling allows you to learn what your students are thinking in a very efficient manner. And I’m struck by the number of times where what I think my students are thinking is not what my students are thinking. And so for me, it has been very, very useful tool to center me in the reality of what actually is happening in the classroom. There’s this wonderful book by Derek Braff, he wrote it years ago before the pandemic head, but I still think it’s very applicable. In that book, he describes many, many uses of it. But just in the interest of time here, one way in which I use it is to check for understanding Another way is when I, particularly when there are questions in which I think students might not be so willing to express verbally how they think about something, I want to be able to allow them an opportunity to do that. And then the nice thing about polling is that it can be combined with other things like think pair share peer instruction, or other things that depending on where the poll results, you can take in one direction or the other. So I’m a super, super big fan of it. And if you have listeners who haven’t tried, no matter what your field, I actually highly recommend that you try it. And the best way to do it, it’s just try one or two polls in one of your next classes and see what you learn from it.

John: And it not only helps you understand what students understand or where students are, it also helps students understand what they know, and they don’t know. And it gives them that immediate feedback that would take longer to do in pretty much any other way.

Dan: Absolutely. It also allows them to commit to an answer. So that allows them to more actively participate. The other thing that I find is that I think it emboldened some students to participate. If they my response has 30% of people who voted for the same response, then there’s something here that I’m not going to be the only one defending this use. So I’m going to go out and defend it. So I’m a big fan of it.

John: And once they commit to that, and you tell them they’re wrong, they want to know why. And that’s not something we always say that committing to that answer is really effective. It is it is Dan Ariely. And one of his books talks about a similar experience where he said he presented these results that he did not find very intuitive. And he gave a talk at some firm. And people would say, Well, yeah, that’s exactly what we’d expect would happen. And then he started pulling them, because then you actually got to see what happened rather than them saying, Oh, yeah, that makes sense. That’s what I would have said anyway. But once you got them to commit to it, all of a sudden, they were objecting they were discussing, and they were engaging with material. And I know Eric Mazhar gave a talk at our campus. I mentioned this on past podcast, but he basically asked people to make a commitment deciding what happens to the hall and a plate of metal when you heat it up. And he went through that whole process where students voted, then they discussed it. And then they voted again, he started to go on to the next topic. And people were angry because they wanted to know the answer at he’s used this example at many places. But one of the things he said is, if I was to give a lecture on what happens, the whole and a plate of metal when I heat it up, it would be about the most boring lecture that you could imagine. But now you all want to know the answer. And so motivating curiosity through these types of things and having that engagement and discussion is a really powerful technique.

Dan: There are several components there, right one is questioning for teaching. The other one is, as you said, the commitment that comes from the poll in the other one is the wanting to know aspect that the whole experience created. The other thing about using poll through technology and about paper that we published some time ago with a student Josh Yardley, and one of my mentors at the Kennedy School Richard Zack Houser, where we compare voting outcomes when students voted by raising their hand versus with at the time, we use this clicker devices, and we discovered big differences in the raise of hands versus the polling devices. So I think another advantage of polling electronically is that they tend to reveal more truthful and it sounds like your story from Dan Ariely reveals that more truthfully how tos actually think

Rebecca: it’s probably that anonymity behind the technology, you have to raise your hand. Now everybody knows whether or not you’re right or wrong. Exactly.

Dan: And also, you are seeing other hands being risen. So you might want to side with the majority in a way that you wouldn’t have if you had to do it electronically. I think

John: this is one of the areas where it seems to take me longer when I’m using zoom. Here, we have a campus adoption by clicker, but this works with any type of boring software, students vote on it, then I send them to breakout rooms. And it takes just a bit longer to do that, just because of the time it takes them to go in and out, then it would in the classroom. So I’m not able to get quite as many clicker questions. And so I have to choose them perhaps a little more carefully than I do. And I’m not using the think pair share quite as actively as I would in a classroom, because it’s really easy to say find someone nearby who has a different answer and debate it for a few minutes. And you can pretty easily see when it’s done in breakout rooms, it’s a little harder to do that. So I generally will, depending on the type of question, I’ll pick the time, which should be enough for everybody, but just the time it takes to get them there and back just adds a little more overhead. But on the other hand, I think it’s still working really well. And maybe by being more judicious in which questions I’m asking that might compensate for the additional overhead costs? And

Dan: I’m not sure yes, and it’s interesting you say this, John, because I have had the same experience. And I wonder if one of the drivers of this is that in a physical classroom, the students tend to know the students who are nearby. So by the third or fourth time you do this, they don’t have this awkward bore you. And my sense is that while the default, and probably a good default that we use in zoom to assign students is random. My sense is that part of what’s driving students taking more time is that they’re often put in this breakout room with someone they’ve never met. And the degree to which they can collaborate quickly on your question about the minimum wage, or whatever you’re asking them to collaborate on probably is not as good as if they had already interacted a lot with each other. So wonder, it does have some disadvantages. But I wonder if you might gain some advantages for those quick questions to always assign students to maybe not the same other person, because then if they all have the same answer, we won’t work but maybe a group of three, this is something that I’ve been surprised by as well. In writing the book, one of the things that became clear is that students tended to like break out rooms by enlarge. But the two main problems they saw with them, his instructions, were not always very clear. And that I think is on us as instructors. And then the second one is, we didn’t give them enough time. And I think you’re right that in a classroom, you can sort of see when the sound is dwindling down. But in the virtual world is a little bit more difficult. I think if you use a collaborative tool, like a Google slide, or something like that, you would be able to sort of see where each group is. But that’s for longer breakout rooms,

Rebecca: That’s definitely my feature request is being able to have more information about what’s going on in a breakout room, even working on little activities or projects, I teach longer extended classes. So they might be working on a project for a period of time. It’s like if I was walking around the classroom, I would just know what they’re doing. Sometimes I can see their files and depends what they’re doing. But sometimes we’re doing code projects or things where it’s not quite as easy to do that.

Dan: I think if you had a Google slide that you can see, but sometimes the word doesn’t lend itself to Google Slides. But you’re right, I think it would be great if they could signal that we’re about 80% down. Here’s prejudice. But I don’t know maybe there’s a future version which we can pull students when they’re in breakout rooms. That might be one way of see Yeah, even

Rebecca: being able to chat with the breakout room would do that. In my class, we ended up setting up slack. So we had that kind of better chat experience while they were in breakouts.

John: That’s certainly been an issue. I know my students are getting much more adept when they’re working in breakout rooms for a longer period and summoning me for help. But it’s really common to get called to one of the breakout rooms I’m talking to them. And then they got another call from another group. And it looks like I’m not being responsive, but I’m really just trying to finish a topic in one group. And it can be a bit of a challenge hopping from group to group because the communication ability to break out rooms, as you both said is limited at this stage.

Rebecca: I’ve had people jumping back into the main window for help. And that works better because then if another group or something jumps back into the main window, they can see that there’s a queue.

John: At least then it’s visible. One of the concerns that many faculty have expressed is that they’re interacting with students who they will see typing in chat and they will hear their voices but they never actually seen the students because most of the time, they have to keep their cameras off because there’s other family members around them. Or they’re connecting to Wi Fi in a parking lot next to a fast food restaurant, or they’re working on a mobile device with limited data plans. So that’s perhaps more of a challenge for faculty than it is for the students themselves. But a lot of faculty suffered Warzone fatigue, when they don’t actually get to see their students. Do you have any suggestions on how people can perhaps get past that?

Dan: So when I wrote the book, initially, I was well aware that I was trying to be helpful to as wide of an array of instructors as I could, but I was well aware that different international contexts might make some of the recommendations harder than others. I think, for the reasons that you suggested, it is hard, in some context, to be able to sort of say to students, please use your camera. So even in my environment, I tried to note them to using their camera, but there are legitimate reasons why they might not use the camera. But what I have discovered, I think you might have discovered in your own faculty meetings, and so on, is that sometimes the issue is not the kind of issue that you pointed out for bandwidth connection or anything like that, is that sometimes you just want to be listening without your video showing off. And what I understand that I think that what we do in our teaching at the end of the day can be profoundly human. And I find it to be very hard to create a human learning experience. If most cameras are off my standpoint, it feels like no one is there. But it’s not only from our standpoint, right? It’s even from the standpoint of the other students in the class. I’m the first one who understands about zoom fatigue. And so I’m not above sometimes having a camera of so I can take a break. But I do feel like to the extent that we can motivate that, and perhaps try if we can to reduce the stigma, use a virtual background, do whatever you can, I think it’s better. But again, I’m not at a place to say everyone should do it, I just be like, the experience can be more human and more effective. If we had most of our students on video,

Rebecca: there seems to be a peer pressure component to that classes have personalities. And if there’s a lot of people who tend to have their cameras on all the sudden there’s more cameras on and if it’s a class that just shuts down, everybody’s in shutdown mode, and breakout rooms, where they’re talking to each other tend to make more cameras appear on from my experience.

Dan: Yeah, I suppose there’s a tipping point, I once gave a I think it was kind of like a webinar where it was on zoom. And you know how on zoom, you have 49 little squares, and then you can go to the other 49 and go on in the videos tend to go on the first. So I think they were like 400 people. And they were like 12 with a video on. And to me that just was very, very challenging. And then I’m not expecting people to put their video on because it’s challenging for me, but I’m expecting that it’d be better experience for everyone. If we can look at each other.

John: I agree, it provides more of a sense of humanity, when you can actually see people, it’s not just that array of black boxes. I know in my own classes, and I think many people on campus have suggested that there’s been that sort of peer pressure to have fewer and fewer cameras all the time. And I think that’s made it a little more challenging for everyone perhaps to have that same sense of engagement, encouraging it is certainly valuable. I think,

Dan: I wonder if this is just speculation on my part. But I wonder if in those contexts to the extent that one of the reasons driving it is I just need a little bit of time off the screen. I wonder if maybe there will be periods in the class where you don’t think that video being on is this critical, and you designate them as camera off period. And the default is camera on. I’m not sure whether that would work. But that’s one idea that just occurring to me.

Rebecca: I’ve experimented with things like that a little bit, Dan, in my longer classes, because he wants to be on camera for three hours. I certainly don’t. That’s right. So like between a mix of breakout rooms, and then little activities that they might do on their own. We do have periods where it’s Hey, we’re gonna have a conversation now it’d be really great to like see, you win an invitation to turn them back on but then also for doing something that’s an independent activity, like we’ve established what behavior of the default this camera off so that people aren’t staring at you while you’re writing something down or whatever. And I also turn my camera off and signal that now’s a good time to turn your camera off and then turn it back on.

Dan: Yeah, that makes a lot of sense to me. Sometimes we will just need a break in some days, not even a video break. Last semester. I remember there was a class where I can I mean, I cannot feel it in the same way that I feel it in a physical classroom, but I could feel that the students were tired. And so one of our teaching assistants I knew that she thought yoga. And I was like, Alright, I know that this is not going to be the end all. But we’re going to do one minute yoga poses for just one minute to just reset. And even that can bring a little bit of energy to the class. And so my short answer to your question, john, is that I don’t know how we solve that problem. But I do think that it is a better experience if we can to have more students most of the time by video, and I like Rebecca’s gentle now it’s a good time to have the video on as a way of signaling when it’s important.

John: Earlier, you mentioned teacherly as a tool to help track engagement. Could you talk a little bit about teaching?

Dan: Sure. So digital is an application, we created that Harvard University group of us to help faculty teach more effectively and inclusively. When you’re in a class you call on different students. And what you see happening very quickly during the semester is that some voices start dominating the classroom and you don’t even realize it. And then you don’t even realize which voices you’re not hearing from. So the way that teacherly works is very simple. You basically have someone record every time that a student participates. And then the students, all they need to do is to fill in a student profile that information about themselves. And as an instructor, then you get access to the student profiles, which allows you to know more about your students, which allows you to search their profiles to see if there are things that you want to incorporate into your next class. And the other thing that it does is it gives you dashboards, about your participation patterns. And so you can see which students have tended to participate the most which students haven’t participated. And most importantly, you can take action to redress any participation patterns that you want to redress. It’s a tool that’s freely available to anyone who wants to use it, the website, it’s teacherly.me. And you adopt the version, there are two versions, the main version is the one that allows you to have student profiles and so on. And it’s been used at Harvard for the last four years, we have over 100 faculty members using it. And last year, we launched an open version so that anyone anywhere in the world can use it. And we’re very happy to see people from other universities started adopting even people from high schools have people at UC Irvine, we have people ideal, we have people in Chi in Costa Rica, and we have people at different places using it. So if you’re interested, go ahead and try it out. I hope it can be helpful in your efforts. I always cite as an example, the fact that before I started using it, I had 46% of students in my classroom one year identify themselves or female students, and only 36% of the comments in the class were coming from female students. And that was a total shock for me, because I didn’t think that that pattern had emerged in my classroom, but it had and this allowed me to take corrective action, where easy to see where you need to take action. And now I’m proud to say I’m not the only digital user who would say that, but I have at least a gender equitable classroom, I no longer have that pattern that I didn’t even know I have, until I started using it.

Rebecca: We’ll make sure that that link to teach Lee’s in the show notes. And that’s a really powerful way to remind us how much data can actually help us that technology can help us in a lot of different ways,

Dan: knowledge and data can help us my colleague studies, we’re owners and Victoria Barnum are also behind this effort. And many of us will be happy to hear from anyone who’s using it,

Rebecca: That leads nicely into what are some of the things that we can take away from this year of technology exploration, as we hopefully start moving back into physical classrooms?

Dan: So this is a question I’ve tried to give a lot of thought, because there’s an aspect that I think is very natural for most of us, we just want to go back to our classrooms, and all of this stuff of assigning race and all of that, but we just want to be with our students in the space. I think for many of us, there’s so much lost that we felt when that environment was in some way taken away. And there’s so much longing for that environment. Again, having said that, I would say that there is so much that we learned about eating during this time that it would be a pity, if we just go back and only adopt nominal change to what we’re doing. My sense is that most of us will now explore using Office Hours through zoom or a similar technology. My sense is that a lot of us created videos for students to engage before class and we might reuse those videos. And I think that’s all great, and maybe the biggest change is that I think Because of this crisis, a lot of instructors were questioning what they were doing in the classroom much more than they did before. I’m sure you see that in your teaching and learning center. And perhaps that questioning and that rethinking about what they did, will translate back into the classroom. But old habits are hard to die. So I think there’s one risk. My sense is that the risk is that we want leverage enough of what we learn in the online environment. And so here are just a couple of things that I’m thinking would be great to think about. One is that I would love for us to try to reimagine or physical classroom in light of what we learn, what is it that frankly, you’re now saying, Wow, this was better in zoom than in person? And how can you go back to your physical classroom and see if there’s a way to leverage that in your physical room? For me, the things that I’ve discovered in breakout rooms have been incredibly powerful. And I don’t mean, just as zoom breakout room, I mean, how can we make the work of groups visible? And how can I be able to see that and leverage that in the discussion? So there’s no reason why when we’re in a physical classroom, we couldn’t use some of the things that we did with Google Slides or jam board or whatever technology we use collaboratively, and try to leverage them in the same way that we did before. That is one concrete example where it’d be a pity to think oh, no, I did that in zoom because of X or Y. No, I think we can do some of that. I know it’s probably for most institutions is too early to think about it. But I do wonder if there are changes we should do in the infrastructure, both technology and otherwise, of our classrooms that might help us teach more effectively. I don’t know about you, but the fact that in so more teams, so whichever technology use, you had the name of each student in front of you in such a clear manner was super helpful to learn their names. And at the Kennedy School, we use name tents. But those name tents are physical. And I wonder if in the classroom of the future, we could imagine digital ones that had more information than just a student name. By the way, I think a lot of changes should also happen in the online technology. Like why is it that the only thing we see is the student name? Why can we hover and see something about their background or their whatever, it should be overlaying so much information that we currently don’t have very case, those are just some ideas, I’m sure that you have many, many more, and that your listeners have many, many more. But if I could leave with one note of encouragement to all of us is to think about what we learned and what lessons were helpful in the online experience and bring it to the in person classroom, perhaps in the different manifestations, but I still think could be helpful.

John: Those sound like excellent ideas. And it does remind me a while back, one of our first podcast was with someone who is developing an augmented reality app to do facial identification for students in her class. So that way, she would be able to get that sort of information popping up in a physical classroom, certainly, I will miss having all the names visible for each student, particularly when I’m dealing with a class of three to 400. Students, it’s so nice to see the name when they participate right on the box, where you see them speaking in a classroom, it would be a lot harder to remember all those names.

Dan: We’ve been discussing everything about what will we do when we go back to the physical classroom, but certainly one thing that I hope we’re going to do is embrace online learning as part of four ways of being able to teach. And I’ve always been a big believer in the power of blended learning of using each medium to its comparative advantage. And I hope that this puts us in a better position to do better blended learning than we were a year and a half ago, when most of us had not done much of this.

Rebecca: Yeah, many faculty up until this point really hadn’t experienced online learning as a student or a teacher. And so now there’s just a lot more exposure. So those conversations can be more concrete. So we always wrap up by asking, despite the fact that we already kind of asked you this question in a different way. We always wind up by asking what’s next?

Dan: Before I respond to this question, and I should have done this at the beginning. But I just want to thank both of you, john and Rebecca for what you do. I discovered your podcast not too long ago, and I’ve gotten tremendous value out of it just to name a few episodes not to name favorite children or any of that. The episode on using Google Apps was incredibly eye opening and helpful to Even though I have been using Google Apps in my teaching, I still learned it on the episode on statistical simulations was super helpful to me it statistics. And that was a wealth of great ideas. And the episode where you took on the workload issue and how students were perceiving that the workload was greater. I had heard other things on this topic. And this was the best of everything that I’ve heard. So I just want to first just say thank you for what you do, and for the service you provide to the people like me, who are trying to teach more effectively everyday. Thanks, Dan. Thank you, in terms of what’s next, I don’t know. At a more personal level, I want to say that writing this book was a totally unexpected thing. For me, I’d never thought about writing a book about teaching. And this in some way has opened my eyes to sort of another world out there that I wasn’t that much in touch with, and has allowed me to feel a great deal of satisfaction when I hear from someone who said, Oh, I use this in the book. And it was very helpful in my learning. So even though I’m super passionate, and educator, and every time that I see a light in the eye of my student, that’s like the most rewarding thing that I can drive for. I think writing the book gave me a different venue with which to see some, I think positive effect of some of what I was doing. And that was interesting surprise. It’s not like I have three books that are in my queue or anything like that. But I discovered that as an interesting thing. And I’m right now writing another book. Again, I haven’t written anyone before, then this one is not about teaching. And that process has been very, very helpful to me, in terms of teaching what’s next, I would really like to see how we can leverage what we learned during the pandemic to do the best we can to help our students learn. That’s my hope. We’re all of us.

Rebecca: Thanks, Dan. This has been a fun conversation. I feel like we need to follow up in a year and see what happened.

John: I would love to do that thing. And also how many new books come out.

Rebecca: Your turnaround time is really, really good. So between the magic wand and the crystal ball, you have I think your setup well.

Dan: I think that deadline helps quite a bit I have to say but thank you.

John: Well, thank you. We really enjoyed talking to you and we’re looking forward to hearing more

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

168. Synchronous Online Learning

Posted in active learning, classroom community, clickers, edtech, evidence-based practices, flipped class, growth mindset, higher education, large classes, learning, mastery learning, online discussions, online learning, pedagogy, project based learning, reflective practice, retrieval practice, small group discussion, synchronous online
December 30, 2020

The pandemic forced many faculty to experiment in different modalities in 2020. In this episode, we reflect on our own teaching experiences with synchronous online courses this year.

Show Notes

Transcript

John: As we approach the end of a really challenging year, we’d like to thank all of our guests who provided so much help and support to us and all of our listeners, and we’d like to thank you, our listeners, for hanging in there with us. We’ve all learned a lot in 2020 and we’re looking forward to a chance to apply what we’ve learned in circumstances in which there are fewer external threats.

…and now we return to our regularly scheduled podcast.

The pandemic forced many faculty to experiment in different modalities in 2020. In this episode, we reflect on our own teaching experiences with synchronous online courses this year.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Over the past few months, we’ve talked a lot about the pandemic and ways to adjust our teaching. And we’ve talked a lot about online learning, but we haven’t really focused on synchronous learning. John and I both taught synchronously this semester. So we decided that in this episode, we would focus a little bit more on synchronous learning and what we’ve learned about it in our own experiments in our classes.

John: Our teas today are:

Rebecca: I have Scottish breakfast once again.

John: …and I have a blend of spearmint and peppermint tea.

Rebecca: That sounds much healthier than my choice.

John: It’s not my first tea of the day.

Rebecca: This is not mine, either.

John: This is my first herbal tea of the day.

Rebecca: This is my second pot of the day. [LAUGHTER]

John: So, Rebecca, what classes were you teaching this fall?

Rebecca: I was teaching two design classes that are smaller. So I had one web design course that was stacked. So, it had beginning, intermediate and advanced students in it, 25 students, and we met synchronously, but also had asynchronous classes, it’s considered a studio course. So for a three credit course, we have six hours of class time, and three hours of outside work, which is a different balance than maybe other folks. And then the other class I was teaching was a special topics design course, which was smaller, it was about 10 students. And that class was also synchronous, but it was a project-based class, and we worked on two community design projects: one for a project called Vote Oswego, which was a get-out-the-vote initiative on campus, and the second is a project called “Recollection,” which is a storytelling project with adult care facilities.

John: And we do have an earlier podcast on an earlier iteration of Vote Oswego. So we’ll include a link to that in the show notes.

Rebecca: So John, what kind of classes are you teaching? We obviously don’t teach the same thing.

John: I was teaching two classes this fall. One was a large synchronous session with 288 students. And the other was a fully asynchronous section with 60 students this semester,

Rebecca: At what level were the students in both of your classes?

John: These were both introductory economics classes. So most students in the class were freshmen, and it was their first economics course.

Rebecca: So, your classes are much larger than mine, you’re teaching much more younger students or newer students, and my classes are smaller, project-based, and usually junior or senior students.

John: Yes. And there’s certainly some differences in the disciplines as well.

Rebecca: No, that they’re the same. [LAUGHTER]

John: Why did you choose a synchronous mode of delivery rather than an asynchronous mode, or a face-to-face option this fall?

Rebecca: So I chose not to do face-to-face delivery for my own health reasons, I chose to not be on campus for my own safety, because I have a chronic illness. So I chose specifically to have strong synchronous components, because a lot of our students are used to working in a studio together and having a community around each other and kind of feed off of each other’s work and work collaboratively. And I wanted, because of the classes I was teaching, to continue to have collaboration as a key part of my class. And I was really concerned that if I didn’t have a strong synchronous component, my students wouldn’t be able to effectively collaborate with each other, because there would be too much scheduling issues and what have you. So it’s a little bit of a carryover from the way that I would run my classes in person in that I give a lot of class time to project-based learning and team-based work and do a lot of lectures and things like that asynchronously in like a flipped classroom style. How about you, John?

John: Basically, I tried to preserve something as close as possible to what was originally scheduled or what was originally planned. And my large class is typically about 400 to 420 students, and I just couldn’t imagine taking that class and doing it in a completely asynchronous manner, because when I teach a class asynchronously, I give students lots of individual feedback, and it would be really challenging providing individual feedback to several hundred students. I just didn’t really have the time to do it in that sort of mode. So I thought it was better to work in a mode where I could give students feedback in a group setting using some interactive tools, where they’re all getting feedback at once. It was the only way I could see handling a group that large. If I was trying to do it as an online class, it would be effectively more in the form of a MOOC with very little interaction, either among the students or between me and the students.

Rebecca: One of the things that we both talked about before we started recording was how we both used a flipped classroom model to help with our synchronous session. So can you talk a little bit about how you did that and what students were doing outside of class.

John: This is actually, in many ways, similar to what I had done in a face-to-face class. Before each class, students would have some readings to work through. And I use the Lumen Learning Waymaker package, which is basically taking materials from a textbook, combining it with interactive multimedia, where they got to shift demand and supply curves and other curves around and see how they responded when they change parameters. And they read a bit in that online text and then they would work through some problems on it where they were allowed multiple attempts at those problems. I also created some videos with embedded questions that were at a somewhat higher level than the textbook readings, which was a little bit more challenging. And they were given unlimited attempts to work through those videos with the questions. And they also, outside of class, participated in discussion boards, where I asked them to relate what they were learning to things in the world around them, in their own lives and their own experiences.

Rebecca: …nice little inclusive teaching practice right there, right? …connecting students with their experience and making it relevant to them.

John: Right, because we know that students learn things most effectively when it has some salience, when they see the relevance to their life.

Rebecca: And the waymaker package, if I remember correctly, had some quizzing and stuff associated with that, and unlimited attempts, a version of retrieval practice there.

John: It’s a mix of things with unlimited attempts and limited attempts. So, the microeconomics Waymaker package is designed, and all of their Waymaker packages, for that matter, are designed, is that they start with a list of broad learning objectives. And they break it down at each module level to sub objectives. And they break those down into sub modules. So, in what would have been the equivalent of a chapter of a textbook, there’s usually two to four sub modules on particular aspects of that. And students work through that. And embedded in it, they have some review questions, some practice questions. And those they can take an unlimited number of times at any point in the course. Once they complete the module, they have a module quiz where they are limited to only two attempts at it. But they’re getting feedback on what they did well. And what they didn’t do well. It’s automatically color coded to indicate whether they mastered the material in one of the blocks of content in there. And then, if they take the module quiz, it will give them feedback on what areas they did well, and what areas they need to work at more. And they’re being directed back to the areas that they need to review. And there, they do have unlimited practice opportunities. And the other thing I did is I created my own videos that focused primarily on the topics that students generally find the most challenging. And in economics, that’s generally with either applications involving math or involving graphs. It was one module a week, and I would take the topics that I know, from past experience, they were likely to have the most problems with, I’d create my own videos with that. And I was using PlayPosit, which allows you to embed questions in there. Most of those videos I created were between five and 12 minutes in length. They would watch the videos and answer questions as they were going. And if they got one of the questions wrong, they could go back and replay that portion of the video and then try it again. And they were given unlimited opportunities for that.

Rebecca: I think you mentioned students really loved those opportunities.

John: At the end of the class, I gave them a Jamboard, which I know is something you’ve used more regularly, asking them what worked well. And there was very much universal agreement on the PlayPosit, as well as on the Waymaker aspects of the course. They really liked the fact that there were practice activities embedded right in their textbook, and that they could go back and try things over and over again until they mastered it. And it was giving them feedback on whether they had, in fact, attained mastery at every step. And it was a nice visual indication of what they’ve learned and what they still needed to work on more.

Rebecca: Excellent.

John: What did you do in your asynchronous components of your class?

Rebecca: Well, the balance of my classes, as I mentioned before, is a little wonky in that we’re supposed to spend more time in class and less time out. So asynchronously, I did a lot of independent stuff that students were not necessarily doing collaboratively. So this is where I had lecture videos that are recorded that were about the topics that they were going to be working on or introduce the component of the project that they were going to be doing. And then they also were completing things like LinkedIn Learning tutorials. And we also have access to D-Q University, which is a set of tutorials for accessibility, and teaches accessibility. So I took advantage of that package as well. And largely they were completing those kinds of tutorials, both of those have exercise files and that kind of thing that they can follow along with. They get little certificates. When they’re done completing there’s little quiz questions and stuff. So they were doing a lot of that kind of work asynchronously. They were also using Slack to communicate with their teams for independent things that they were working on that they needed to communicate out to teams when they were working on projects together. And I also use Slack as a place to have discussion. So like you, I had discussion questions that tried to make what we were talking about relevant. We were exploring design, specifically like web design and how they interacted as a consumer versus how they would interact as a maker and did a lot of observational studies. We also did some discussion boards that were really about design activities and things that got students off the computer. So they were just documenting what they did off-screen, offline. So things like listening to a podcast so that they didn’t have to be staring at a screen and what their takeaways were. They attended virtual conferences, which I guess was still on-screen, and did some sketching, like paper prototyping and some other methods that we like to encourage our students to do, just to kind of help balance the screen time a little bit for students. So that’s largely what they were doing asynchronously.

John: Were you having them submit some copies of that work in some way, or were they just reflecting on the work that they had done?

Rebecca: The little non screen activities were documented in a discussion, essentially, that we were holding on Slack, and then tutorials and things, they were just submitting their completed certificates. And so I broke down those LinkedIn Learning courses and things over multiple weeks. So they didn’t really submit those certificates until they were completed. But they were doing a little bit by little bit, but if they didn’t do the tutorials, they wouldn’t be able to do the projects or the actual work that we were doing of the class. So it was pretty important that they were doing those components outside of class.

John: Once your students were in class, what did you have them do in a typical class session?

Rebecca: The two classes I was teaching I handled a bit differently because of just the sheer volume of students in the bigger class, which was 25 students that were working on projects. They were working on collaborative projects, in teams of three, for the most part. And so what we would often do is show-and-tell’s or critiques in small groups. So let’s say there was two or three teams together that we would do a little critique with in a breakout room, while other teams were meeting and collaborating. We would also do things like come together to answer questions about things that they were working on, troubleshoot or whatever, and then go work on projects in breakouts for a bit. And then we’ll come back at a scheduled time. I also did one-on-one meetings with students during class time. So I’d set up things like a quiet work breakout room or the chatty breakout room. And students would pick the place that they wanted to go while they were working on projects. And then I would meet with them individually for critique, and often a lot of code troubleshooting is a lot of what I spend synchronous time doing. And students sometimes met with my TA to do the same thing, and with our small groups. I also did a lot of design challenges. And students really liked those and would like to do more to hold them accountable for the kind of material that they were learning outside of class or being introduced to outside of class in a low-stakes environment to test it out with some peers and troubleshoot. So I would pose a little design problem. And then they’d work in a small group to work on that problem in a very tight amount of time. They might spend 30 minutes… my classes are three hours long… or an hour, and then we’d come back and show them off or talk about different things. And I tried to make those design challenges fun and entertaining. So one of the first things we did, which worked really well to start gelling their teams that they were with the whole semester was designing an emoji for Slack that they used for their team. And they loved that assignment. It was partly about working at a small size, and so it was tied to some of the curriculum that we were doing, but it was fun. So they did that in a small team and then had to implement it. Later on in the semester, we did things like a 404 error page for their projects, which were just kind of entertaining. We tried to make them amusing, so that if you landed on a page, it was a good user experience. So things that maybe wouldn’t typically work on in one of my classes that were a little bit more fun, but really were emphasizing the technical and conceptual things that we were working on. The other thing that we use synchronous time for is I took advantage of our virtual platform, and I brought in alumni multiple times, and local designers multiple times and did little Q and A’s with them. Not every week, but every few weeks, or every couple weeks, I would bring in a designer for a 30-minute session. They’d introduce their work. And then students did a Q&A with them, which students really loved. And it broke up our time a bit and really gave them something special that maybe we didn’t always do in a face-to-face class that made the synchronous environment kind of special.

John: Excellent. That is a nice opportunity provided by Zoom that actually could work in the classroom too. But I think many of us just hadn’t really considered it so much. It doesn’t really matter where you are when you’re teaching in this sort of synchronous environment. So it’s very easy to bring in guest speakers and it’s something we’ve probably should have been doing more of in the past, but I think many of us will be doing more in the future.

Rebecca: So John, how did you use your synchronous time?

John: I had told students before each class session, what specific topics we’d be working on. And then most of the class time was spent asking him a series of problems of progressively higher levels of challenge. I basically adopted Eric Mazur’s clicker strategy of trying to find challenging questions where roughly half the class will get it wrong the first time and then letting them meet (in this case, I had the meet in breakout rooms), discussing it and coming back and voting again on it. And generally, you’d see a fairly significant increase in the performance after they’ve had that chance to engage in peer discussion. And that’s where a lot of the learning seems to happen when clickers are being used. I used iClicker. The only difference is students could not use a physical radio frequency clicker because they have a range of a couple 100 meters and students were spread out all over the world, I had one student in Egypt, I had students in South America and students spread throughout the country this time. So they needed to use either their laptop or a mobile device in order to do that. We discussed it as a whole class after they come back from the breakout rooms. And then I’d asked them to explain their choices. I generally have them use chat, and then I’d go through and correct any misperceptions they’d have. And I try to guide them to the correct answer by asking them questions, and letting them see for themselves why some of the answers were right, and some of them were wrong. And generally, that’s how we spent many of our classes. Initially, I was also using Kahoot! from time to time. They enjoyed Kahoot!, but I noticed a bit of a drop off when we were doing the Kahoot! sessions, because those were not graded. And with the clicker questions, they were being graded, and that tended to receive a somewhat higher level of interest. It was very low stakes, they got a certain number of points for an incorrect answer on either attempt, and they got a bit more points when they answered the question correctly. And initially, I was giving him three points for an incorrect answer, and five for a correct one. And they asked it perhaps that could be bumped up, because some of the questions were so challenging. And I did raise it. So they ended up getting four points for any answer, and five points for a correct answer. So it is extremely low stakes. So I tried to do a lot of retrieval practice in the class, where it started from essentially no stakes with the embedded questions in the reading, then it ramped up to in class applications of this, where they still get 80%, even if they got it wrong, but they had another chance to get it correct. And then they took that module quiz, and even there, they had two attempts at it. So if they made mistakes, they had lots of resources they could go back to and work on it. So I tried to set it up and provide them with many pathways to attain mastery of the content, and to encourage a growth mindset and to encourage them to recognize that people make mistakes when they’re learning and that there’s a lot of benefit from having those mistakes as part of your learning process. There’s a lot of research that shows that we learn things much more deepl if we get them wrong, when we first try it, we’re much more likely to remember it later on, then if we happen to get it correct, initially. In that case, we’re much more likely to forget it a bit later. And that was a bit of a challenge for students. But I think they eventually appreciated the fact that everything was fairly low stakes.

Rebecca: I think I’m seeing some themes in the things that, although we’re teaching very different classes in very different contexts, there’s some real big themes about how we’re using our synchronous time, and even how we’re using our asynchronous time. And so there’s an emphasis on peer interaction and establishing those peer networks, really enforcing or reinforcing things and dealing with muddy points. And then also just providing the encouragement and support like that low-stakes environment or trying to foster a growth mindset. So in my classes, I did the same thing. I was doing peer group work and trying to really get them to collaborate and troubleshoot together and they love that that… that was really valuable. I spent time doing live demos and troubleshooting, when there was a really troublesome technical component or something that they were trying to do that a lot of them were having trouble with, that they could ask me live questions. So that same muddy point kind of thing that you were getting to in what you were discussing. And then, finally, the growth mindset that you started bringing up, I’d also tried to do and, although I didn’t have a lot of low-stakes testing, or something like that, I set my projects up so they were done in sprints. So a long full-semester project was broken into multiple two-week sprints, where they would work on something, get feedback, and then could revisit whatever they did, and then add a new component to it. And so I did that throughout the whole semester. So there was a bit of retrieval practice, a bit of spaced practice in there, and certainly some fostering a growth mindset and the idea that you make mistakes and that’s how you learn. And I spent a lot of time… I don’t know if you experienced this too, John… but I experienced a lot of time in synchronous and saying like, “You can do this. It’ll be okay. And this is how the learning experience works.”

John: And I did have to do a lot of that, especially in the first few weeks of the semester, because they were not used to a flipped class environment. And they were not used to this notion of making mistakes and learning from mistakes as part of your learning process. Because most of them have come up through their elementary and secondary school system thinking that they need to memorize some things and reproduce it on exams. And they do well if they get high scores, and they don’t make mistakes. And that’s just not how we learn in general. And it was important, I think, to help remind them of that. Another aspect of the flipped class environment that we’re both using is that we let students learn some of the basic skills, the easy things that they can learn pretty easily on their own, from other resources. And we’re trying to focus our class time using essentially a just-in-time teaching approach where you focus on the things that students always have trouble. In a traditional classroom environment, what normally happens is students will learn the easy stuff in class where faculty will lecture them on basic definitions and basic concepts. And then it all makes a lot of sense until students try to apply it. And they try to apply it typically in assignments outside of class, or in high-stakes exams. And it’s much more productive if the students use the time outside of class to master those basic concepts. And then we hold them accountable for having done that somehow in class. And then we give them assistance on the things that they find challenging when they need it. Not after they’ve had that experience of a more high-stakes assessment in some way.

Rebecca: Yeah, I think what I found or that students really shared with me that was something that they really appreciated was that there was a lot of structured time to work on those difficult problems in class. This is true of my face-to-face classes too, but even maybe more so in this online environment where students were having a really hard time managing their time. I would allow time to work on a project during class… it was scheduled, but then there was a check in point later on in the day. You wouldn’t want to spend three hours staring at a screen on Zoom, like this makes no sense. So I certainly did not do that. And I don’t want anyone to think that I did that. But, I would do things like “Okay, we’re going to check in at 9:30. And then we’re going to do a little activity together. And then you’re going to have some work time to work on X. And then we’re going to come back at 11. And you’re going to show me what you did. And then we’re going to have a little discussion or do another little activity, and then we’re gonna come back again at 12.” And we would have a schedule where there was time to kind of come back. What I found is, over time, students often wouldn’t actually get off of Zoom. They would just turn their cameras off and their microphones off. And I would do the same if it was like a work time. And then when we all came back on, a lot of students would turn the media back on. That said, I, of course did not require that depending on where students were, I certainly had students that were in environments where they couldn’t turn their cameras on, or had really poor internet connections, we adjusted as necessary there, and we had a way to communicate in a much more low-tech fashion using Slack during class time. So if something happened with someone’s internet connection, or whatever, they could still stay connected with us and what we’re doing.

John: How did you assess student learning in your class?

Rebecca: My classes are all project based. So the majority of grades are built on projects, not entirely, we had discussion boards, and I had some collaboration things that they were doing, and they were evaluated on those things as well. But projects were the significant piece of the puzzle. And the way that I graded them was really just providing feedback about the kinds of things I was going to ultimately grade very regularly throughout the semester. So every couple of weeks, they were getting feedback on their code for my web class, for example, feedback on their design, feedback on their writing, not a specific grade, necessarily, but feedback on all of those elements that were going to go into the final project. And then the ability to revise all of those again and again and again and continue to get feedback on those.

John: Did you have your students engaged in any reflective tasks?

Rebecca: Yeah, that’s a really great question, John. I had reflection built in two ways. So at the end of each sprint, or kind of module in my class, they were working on two projects at the same time throughout the semester, so they’d work kind of two weeks on one project, two weeks on the next project and cycle back… that was so that I had time to give them feedback regularly. So that was part of my structure. But at the end of one of those modules, I had a reflection activity that I implemented using a Google form. So a few different prompts to think about what they got out of that sprint, goals for their next sprint, that kind of thing. And then I also had some big group reflections at different moments during the semester, I had one at the beginning, and a couple in the middle and one at the end. And I use Jamboard for that, which is a Google suite tool that has sticky notes, and is the same kind of way that you might brainstorm. So I use it as a way to collect reflections in sticky note form, essentially, virtually. And I would have a reflection question for folks to respond to or a couple of different boards with different kinds of questions. In the beginning, we did something called “hopes and fears,” which is something I’ve talked about before… setting up the class like, what are they hopeful that they’re going to get out of a collaborative project? What are they scared about? We find out that like, all the teams have the same hopes and fears. During the middle of the semester, what are some of the big takeaways that you’ve had? What are some things that you want to work on? What are some things that you’d like to see changed about the class and various themes bubble up on that. And then at the end of the semester, I asked questions like, “What was your biggest takeaway? What was the thing you were surprised that you learned? What is one recommendation of something you would change in this semester?” and “What is something that you want to continue learning?” and I got really useful feedback on what to change about the class but also, some really great themes bubbled up across the class, which really results in like kind of three or four things for each of those questions, which was a nice way to wrap up the end of the class and summarize for students after they completed that task. And one thing that I like about the Jamboard is that it actually ends up being anonymous. You can see people while they’re working on it, but it doesn’t keep a name with a sticky, ultimately.

John: So you can see who’s active in the board, but you don’t see who is writing which note.

Rebecca: Yeah, exactly. So that worked really well for me. How about you, John, were you able to build in reflection? I know you have such a big class. So it can be tricky.

John: I wasn’t able to do as much of that with my large section. But I did have them do that, to some extent in their discussions. For one discussion forum in both classes, I had them use a tool called Packback, which uses artificial intelligence to give students some feedback as they’re writing their prompts. And each week, students had to post a question related to that week’s material and they had to respond to at least a couple other people. But one of the nice things about Packback is it will check the cognitive level of the posts, it will give them some feedback in terms of grammar, it will also do a little bit of checking to see if the material has previously been posted. And it gives students some feedback, encouraging them to say more than “I agree.” And it also encourages them to document sources and to provide resources or references for the arguments they provide. And they get a score on that. So it takes a lot of the evaluation of that away. And so I monitored all of that. But it was something that seemed to function pretty well, just by the interaction between the users and that system. I haven’t really mentioned much about what I did in my online class. My online class uses many of the tools, but obviously, I couldn’t do synchronous, because that class is fully asynchronous. I couldn’t do the same type of instruction. But I had students do two other things in that class that provided opportunities for reflection, one of which was I had them work in a metacognitive cafe, low-stakes discussion forum, where they reflected on what they were learning and the learning process. And that gave them another way of making connections to their learning and reflecting on how well they were learning materials and what barriers they were facing, and also sharing effective learning strategies with each other. They were given some readings each week, generally on research-based learning practices as a primer for many of those discussions… others, they were just reflecting on what they’ve learned and how it might be useful in their life to tie it back to themselves. But the other thing I had students do is work on two podcast projects in that class. And in those they were taking what they were learning and making reflections about how that connected to the world around them. Many of them ended up being related to COVID and pandemics, but they were making some really good connections, and they were getting a chance to see how the material they were learning had some relevance in their own lives. And a lot of that came out in some of the things they were discussing in their podcasts. And they also did use Jamboard once at the very end of the term. But I also use Google forms a few times to have them reflect on the process of what was working, what wasn’t working in the class and what was working and what was not working in their own learning processes and what I can do and what they could do to help them learn more effectively.

Rebecca: I don’t know about you, but I was really surprised at how well synchronous learning actually went for me. I had some technical difficulties early on with my internet connection. It took me a while, but I got around to fixing that problem by hardwiring my internet and resolving some of those things. But I felt just as connected to my students as I would normally. I had a lot of interactions. And in some ways, I was able to facilitate those interactions a little more equitably online, because it wasn’t just the person who came to nudge me and stand in line and be the next person. Instead, I could really coordinate using waiting rooms and breakout rooms and really give everybody a chance to have one-on-one interactions with me, which I really appreciated. And I really did get to know all of my students quite well, which I was a little bit surprised about. And then, in an area where it’s really technicalaAnd we’re doing a lot of coding and things on screen, being able to share screens and take control of another person’s computer to fix things or show them how to do something was incredibly valuable. We use some of those kinds of tools in person. But it actually was, I think, in some cases more effective using this particular tool. So I was kind of surprised at how well some things worked. And I think that even when things go back to face to face, there’s definitely some components here that I would keep.

John: I’d agree. And I think students were amazed at how well some of those tools work. When in breakout rooms, they would be using the whiteboard features, they would be sharing screens, they’d be making the case, they’d be drawing on the screens, and that was something that would be much harder to do in a face-to-face environment. Initially, at the beginning of the class, I had some issues with chat being kind of flooded with irrelevant material, and I had to clamp down on that a little bit. But within a couple of weeks, they started actually using it very productively, and it provided a voice for all students, even those quiet students who would have otherwise sat in the back of this large lecture hall. They were able to type something in chat, after thinking about what they wanted to say before doing it, without being concerned about interrupting the discussion that was going on. And I think that was really helpful. And when I taught large classes with three to 400 students, there’s almost always 3 to 10 Students who have trouble not having side conversations when there’s other activities going on. And that mute option is kind of a nice feature and the ability to set their microphones so they’re all muted unless they choose to unmute… to have the default being muted until people click the unmute option… made it really easy. And I was amazed at how quickly they adjusted to muting and unmuting. By the end of the term there was maybe only once or twice a class where a family member or someone else would walk into the room and start talking. And then they’d remember, they had to mute their mics, and it was very rare. In a class that large, I was impressed by it… and working with students one on one, during office hours, it was so much easier to have students just share the screen and show you exactly what their problems were then to correspond with them with email, or even have them boot up their computer or you try to find what they were talking about when they came to your office. It was just much more efficient.

Rebecca: Yeah, I could actually see it. You can Zoom in, you can see what they’re talking about. I also found, and I was really floored, in this last week of classes, students were doing their final presentations,at how well they did develop facility with these tools. They’ve developed a lot of fluency in the kinds of tools that are actually very relevant to my particular discipline. It’s relevant to many disciplines. But designers use these tools all the time when they’re working with clients. And so it was amazing to me that we got through 15 presentations so efficiently. We didn’t wait for anybody to share their screen. They just knew what they needed to prepare, had it ready, they started developing slide decks really effectively, and could just do the things that they needed to do really efficiently. One of the last things I said to my class was like “I’m so proud of you just being able to do that. We didn’t have to wait for anybody today. That was amazing.” And so maybe a little bit of a blessing in disguise, you hate saying like, “Oh, the plague is such a great thing.” But they really did develop some useful skills and tools and they became more effective communicators. That was something that a lot of students reflected on and things that they didn’t expect to learn is how much better they became collaborators and just communicators generally… not just in person, like through Zoom or in text… like through chat in Slack.

John: Video conferencing is likely to be a part of their lives in the foreseeable future, especially now that everyone has adapted tp this mode, it’s very useful for them to learn how to use that efficiently. The one thing I do miss though, is seeing their faces in person and recognizing them. One concern that I have is, I’m hoping to be back on campus in the fall, there may be students that I work with who interact with me regularly, whose voice I would recognize or whose name I would recognize on the screen, but whose face I just wouldn’t recognize because a very large proportion of students just didn’t feel comfortable having their cameras on regularly, and I understand that. We’ve got a lot of students living in crowded living quarters or working with very poor network connections. But I do miss actually physically seeing them. And I had my last class session earlier today. And I encouraged them to stop by in the fall and just say hello.

Rebecca: Yeah, I mean, I agree that the physicality is certainly something that’s missing. But it was amazing to me how connected I still felt to all of the students at the end of the semester. And I think that they felt connected to each other too and they verbalized that, and also wrote that in their Jamboard reflection. So although there’s much to be improved, given this was the first time out and an experiment in many ways. I’m really thankful that I read Flower Darby’s book about Small Teaching Online because that actually informed a lot of my practices, even though it was synchronous, and a lot of her material was about asynchronous learning. It really did help me remind myself of things that I already knew that I needed to do, but to kind of make a checklist of things that I definitely needed to do as I was rethinking my classes for the fall. So thanks for chatting with me, John.

John: We always end with the question: “What’s next?”

Rebecca: I am sitting down to reflect on what worked and what didn’t work, to try to troubleshoot some things for the spring. And I’m teaching a class that’s brand new to our curriculum for the first time in the spring. And so we’re developing it for online synchronous, although ultimately, it’ll probably be a face-to-face class. We’ve had to re-conceptualize some of the things that we were going to do because of the technology limitations that students may have. If they’re online, we’re expecting that we might have a lot of students who are relying on their phones versus software and having access to high-end software packages or computers that can run them. So we’ve had to rethink things. But I’m pretty excited about being able to experiment with my students with all kinds of technology in the spring, but it’s definitely a puzzle that I’m currently starting to work on. How about you, John?

John: Well, I still have a lot of grading to do. But once that is done, one of the things I’m going to be doing is converting a textbook I had written in econometrics to a Pressbooks site, which will be a lot of conversion because it’s originally in LaTex, a typesetting language used for mathematical typing and I’m planning to create a lot of videos, I’m hoping to get many of them done over the break so that I’m not spending 15 or 20 hours a week creating videos as I was all fall. And I’m hoping to get a little bit further ahead of the semester this time, so I’m not doing as much preparation at the last moment. And we’re both going to be working on putting together a series of workshops in January for our faculty to help people prepare for whatever comes at them this spring

Rebecca: We’re just going to be really busy.[LAUGHTER]

John: I’ve never spent as many hours working on my classes as I have this semester.

Rebecca: I agree. There was a lot of startup costs converting to this modality, but I’m hoping a lot of that stuff I’ll be able to keep and reuse moving forward. Thanks, John. Always nice talking to you, John. We chat all the time. But it’s nice to sometimes hear about some of the thought process and things behind some of the decisions that you’ve made in your classes. So it was really nice to actually hear about how you did some of that stuff this semester. So thanks.

John: And I also appreciate hearing more about what you’ve been doing in your classes. We spend most of our time on podcasts talking to our guests and only mentioning little snippets of what we’ve been doing ourselves.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

153. Structured for Inclusion

Posted in classroom community, clickers, equity, evidence-based practices, first-generation, higher education, inclusion, inclusive pedagogy, instructional communication, learning, learning spaces, pedagogy, social justice, teaching
September 16, 2020

Learning spaces that are effective for all students require careful planning and design. In this episode, Viji Sathy and Kelly Hogan join us to discuss ways to promote inclusion in the way we structure our courses, activities, and feedback. Viji is a Teaching Professor of Psychology and Neuroscience at UNC – Chapel Hill and Kelly is an Associate Dean of Instructional Innovation, Quality Enhancement Plan Director, and Teaching Professor of Biology, also at UNC – Chapel Hill.

Show Notes

  • Eddy, S. L., & Hogan, K. A. (2017). Getting under the hood: How and for whom does increasing course structure work?. CBE—Life Sciences Education, 13(3), 453-468.
  • Poll Everywhere
  • Hogan, K.A. and Sathy, V. (forthcoming, 2021). Embracing Diversity: A Guide to Teaching Inclusively. WVU Press.
  • Hogan, Kelly A, and Sathy, Viji (2020). “Optimizing Student Learning and Inclusion in Quantitative Courses.” in Rodgers, Joseph Lee, ed. (2020). Teaching Statistics and Quantitative Methods in the 21st Century. Routledge.
  • Panter, A.T.,; Sathy, Viji; and Hogan, Kelly A (2020). “8 Ways to Be More Inclusive in Your Zoom Teaching.” Chronicle of Higher Education. April 7.

Transcript

John: Learning spaces that are effective for all students require careful planning and design.
In this episode, we discuss ways to promote inclusion in the way we structure our courses, activities, and feedback.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guests today are Viji Sathy and Kelly Hogan. Viji is a Teaching Professor of Psychology and Neuroscience at UNC – Chapel Hill and Kelly is an Associate Dean of Instructional Innovation, Quality Enhancement Plan Director, and Teaching Professor of Biology, also at UNC – Chapel Hill. Welcome.

Kelly: Thank you.

Viji: Thank you.

Rebecca: Today’s teas are:

Kelly: I’m drinking LaCroix… seltzer.

Viji: Yes, me too. I’ve got my sparkling water right next to me.

Rebecca: That’s my second favorite thing to drink, over tea. [LAUGHTER]

Viji: In our writing last summer, we would get together, when we could get together. We would get together and write, and we often had a nice cold sparkling can of LaCroix with us, and one time we tweeted about it and LaCroix contacted me and sent me some water. So…

Rebecca: Nice…

Viji: …it’s become our official working drink.[ LAUGHTER]

John: Somehow tea has for us, as well. I have blueberry green tea today.

Rebecca: I have “Special” English Breakfast tea.

Kelly: What makes it special?

Rebecca: The package? [LAUGHTER]

John: The label? [LAUGHTER] Okay, and where did that come from?

Rebecca: It’s a Harney’s and Sons tea.

John: You’ve both been working together for quite a while now on inclusive teaching practices and have done a really good job in providing lots of workshops and lots of materials for people who would like to improve their teaching practices. What prompted your interest in this area? And how did you start working together on this?

Kelly: For me, I think I started getting really interested in what it means to be a good teacher based on data. So I had seen some data in my own course. And I saw some pretty large discrepancies based on race and ethnicity. And I thought a lot about what it means to be effective. And it really got me thinking about: are there ways that I could narrow and reduce those achievement gaps in my own class? And not long after that, I was in a faculty learning community for teaching large classes, and that’s where I met Viji. So, we were both in this faculty learning community together, paired up in a group, and we quickly recognized ourselves in each other. So, just our style of teaching, our personalities are on the more introverted side, we recognized that we really enjoyed learning how students learned, but weren’t always going to be the most charismatic and funny people. And we felt really strongly that funny didn’t equate to good teaching, and so we really built a friendship and collegiality around really learning, with each other, what good teaching looks like.

Viji: Yeah, and I’ll add that we had the opportunity, in that faculty learning community, to watch each other teach. And up to that point, the only time I had been observed was really for what I deemed sort of high-stakes purposes, like for renewal of my contract or something like that. So this is the first time we got invited to just sit in a classroom for no other reason than to just see how another instructor operates in that classroom and it was very eye-opening experience, because not only was it a chance to do this without sort of a weight around it, but also that it wasn’t a topic that I didn’t know anything about. So, it became a really fun activity to sit in the classroom and just be a student and see it from a student’s perspective. And especially not knowing the content, specifically, it was not about critiquing the content or the delivery of the content, it was really just the mechanics of teaching and what that looks like. And that was a really helpful thing for me to see and experience being a student in Kelly’s classroom.

John: Is that something that was done for just people within the learning community, or more broadly throughout the institution?

Kelly: Those observations were part of the faculty learning community. We have since tried to build programming around that same idea, campus wide. And so we have a peer visits program that we help the Center for Faculty Excellence run and faculty can go into other people’s classes, they can see a menu of people that are available that they can go visit, some rubrics available. So, I think it’s spun out of that, as something really transformational for us that were involved early on.

John: We were just planning to introduce one of those beginning in late March of this year. And then it kind of fell apart because people were no longer interested in doing that when they were panicking in terms of the transition to remote teaching. But, we’re going to be meeting next week to talk about how we might be doing that here. So, it’s something that I’ve been encouraging… I’ve been trying to get some motion on for a while now. And it looks like we’re moving in that direction. And it sounds like it was a really productive experience for both of you. And for the rest of us, given your collaborations since then. Many people have been concerned about the growth in income inequality, and economists have done a lot of work showing that one of the main reasons for that is the growth in the rate of return to education over the last few decades. What we’re seeing are some very unequal outcomes, as you mentioned, in terms of success in courses, persistence, and so forth by race, and in the STEM fields, also by gender. So, it’s really nice to see people working in this area, because it’s an area where I think we need a lot of help. To what extent are they These differences that we’re seeing the result of systemic racism and sexism.

Viji: There’s a lot in that question. Well, racism, sexism, any form of discrimination… In essence, these are learned behaviors, and these are things that we grow up with without really even thinking about sometimes. And the classroom is no different from being in life. And so we have to address them in the classroom in the same way we need to address them in life. And for me, when I think about it, it’s really about sort of concrete things sometimes, like who is speaking up in a certain space? like who feels comfortable speaking up? Who feels comfortable speaking without really having much time to think about their answer? Who gets to see instructors who look like them in the classroom? We already know that, especially for our students, it can be difficult sometimes for them to identify with their instructors, to feel like they’re just a normal person. Sometimes we hear that, right? Like “You do the things we do? That seems so strange. I never would have thought a professor would do those things.” Right? So even identifying with a professor, like adding that layer of seeing somebody who looks like you in the classroom just makes it feel even more unattainable, right? So, there’s a lot in thinking about a lot of aspects of teaching that are barriers for our students. And I often, when I go to a professional conference… when I was able to go to professional conferences… I looked out into the room and what I see in my professional meetings doesn’t look like what I see in my classroom, in terms of the diversity of participation, and I asked myself why that difference exists. And my course is the first course that leads people on a path in what’s called quantitative psychology. So, if I want them to have more people, more interested people, in the field, they have to succeed in my class to then have the interest and the goal to keep going on that track. So, it starts with my class, but it actually starts way before my class and all the messages they get before they even show up at my doorstep in my course and how I can work to counteract some of the messages that tells them they don’t belong, and that there isn’t a place for them in STEM. These are things that they hear either subconsciously or consciously and we need to address that.

John: What can we do to create a more inclusive learning environment for our students that will work well for all of our students?

Kelly: Well, I think we have to recognize that these historical differences, as you said, systemic racism and sexism, that those are things that existed before we met our students, and they lead to differences in who our students are. But, we have to be careful not to blame our students for those differences. You know, diversity is a strength, and we have to find ways to feel empowered to work with the students that we have, to build on that strength that is the diversity, but also not, as I said, blame students. So, the way we like to think about this is by adding structure to everything that we do, and we like to think about it as structure in the course design as well as the facilitation in live sessions. So, a lot of times our students, especially, see teaching is just what we do sort of face to face or in this day and age our live zoom sessions, if we’re doing them… and who’s not speaking up and who’s not participating if we only use low structure, and by that, I mean, like maybe one mode where we expect volunteers all the time. But, we also have to think about course design and a low structure course design might be one that doesn’t have a lot of practice and assessment built in, where students actually learn how learning works. And so we want to think about building structure in everything we do, and asking ourselves constantly: “With what I want to do, how can I add more structure so it’s not left up to chance. Who’s going to know what to do with this? Who’s going to know how to take notes? Who’s going to know that there should be routine practice in learning? Who’s going to know that they could participate in different ways? So, that’s kind of the way in which we think about it, but I’m sure we could get into more specifics with each of our courses.

John: And you’ve both done some research that have shown that there are significant effects of providing that structure in terms of encouraging student success, as well as perhaps reducing that gap, I believe.

Kelly: Yeah, absolutely. Work that I published with a colleague, Sarah Eddy years ago, we looked at my teaching in a much lower, less inclusive structure where I did a lot of talking… you could call it a pretty typical teacher-centered classroom… and then looked at three semesters of me shifting to something far more student centered, a variety of ways of interacting with my students, and basically a higher structure classroom. And even in those first few semesters where, you know, you’re just getting used to something and don’t feel proficient yet, it made a big difference. It closed an achievement gap for first-generation college students, it narrowed the achievement gap for black students, continued to see students talk about an increased feeling of community, among other things. So, it continued to get better as I got better. And I continued to see ways I could put more structure into my course. And I kept asking myself, how can I add more structure?

John: Maybe we could talk a little bit about some of the ways in which you’ve added that structure in each of your classes.

Viji: Sure, I’ll provide some examples of that. When I redesigned my course, and, like Kelly, I had landed as a study to look at how, at that time (it was about maybe 10 years ago), recording micro lectures and having students watch them before they came to class, and using class time to do more polls and some of the assignments that they were struggling with. And that was the challenge I had in my introductory statistics course was I was using the class time to explain ideas and then sending them home to do hard homework problem sets. And oftentimes, that led to a lot of frustration, because there was no one around to help with the questions that they had in real time. So, I wanted to switch the order of that so that they were watching the videos where I’d explain how you might calculate something at home, and then in class, we might practice doing some of those problems together, with peers, with graduate students, with undergraduate learning assistants. So, that’s an example of a structure that’s in place, right? …having the videos available so students can watch them before class. And what I learned was it became a really incredible resource for students to access throughout the semester. I anticipated that they would get used right before the class session where we’d be using the material. And indeed, when I look at the learning management, the site provides some statistics around that. Yes, there were the most clicks right before class, because I had a quiz in class that day on that material. But there were also clicks right before that first exam on some of those videos. There were also clicks before the final exam. There were clicks in random days in the semester when I didn’t think it had anything to do with what we were talking about. But, they went back to watch something. And what that taught me was that they need to see that material more than one time. And when I was doing it in class, it was once, it was ethereal, right? …it was once and it was gone. But, now students could rewatch, they could hit pause, they could work as slowly or quickly through the problems as they wanted to. So, it provided a resource for being able to do that. And again, that’s the example of, by providing it, not all the students need to watch it multiple times. But, it’s available to those who needed to do that or wanted to do that.

John: So they’d watch a video and then you had a quiz at the beginning of the class or was it before the class started?

Viji: The way I implemented it, and there’s lots of different ways people have this piece, how they would structure that requirement, but I wanted it to be done. And so I wanted students to have shown me that they’ve done it through a quiz at the start of class. It helps keep them accountable for doing the work. And I do a fairly good job of what we call, it’s like “the warm demander.” I’m the warm demander in the classroom, and I do a pretty good job of coaching them and asking them to do this work so that we can do hard things together in class, making the argument that it’s the most efficient way we can be together, when we’re together. And then there’s peer pressure, right? Like if they’re the only one, they look around, and everybody else came to class prepared. We’ve all been in meetings where we didn’t do our homework, whatever the homework was. So, if you build this culture, I think people really do take to it and they do learn that it is efficient. And more importantly, like in Kelly’s class, they see results, they do better on an exam because they’ve kept up with it all along. So, that’s when you know, the proof is in the pudding, when they see things that they’re pleased with and they keep going with it.

Kelly: Yeah, and that’s an important point Viji just made… that these kinds of techniques help all students, they disproportionately benefit some students, which makes a difference in terms of equity, but they definitely help all students. My own experience with structure is one that Viji alluded to with the flipped classroom, which is another way of thinking about the learning cycle, that students need to be required to do things before, during, and after class. And that adds a very high structure to what we would consider the learning cycle. So, if I ask students to do some reading before class, I don’t assume that all students know what to take away from a reading. And so for this, I give students guided reading questions and it helps them know where I’m coming from and what they should focus on, and what they might want to use as a study guide. And it helps replace the lecture so that I’m not going to talk to them the whole time that we’re together. When we are together, I want to use the time for collaboration and a variety of things. And so, I also don’t assume that students know what to take away from that. And so I provide class outlines, to make sure that, whether a student has learning differences, is multilingual, distracted, whatever, that all students leave with some basic outlines from class. So, already you’re starting to see how the structure can help all kinds of students. And then in class, I added a lot more active learning, and it quickly became apparent to me that, if I don’t put the instructions in multiple modes, so verbal and visual, that students were not going to be with me, and we were going to waste a lot of time with instruction. So, it’s something I think we don’t think about a lot. Like, if we want them to do something, then we have to be very clear about that, whether it’s in an assignment, a breakout group online, or active learning together in a classroom, providing more silence time for thinking. And then, for me, a lot of it has come down more recently to group work and equity around group work. And I kept thinking to myself, how do I add more structure to the group work because students were telling me if I just said, turn and talk to a neighbor, that certain students always were left out or they were with friends and they weren’t being pushed to really do the learning and feel the rigor of what they were being asked to do because the friends would just sort of agree and then chitchat, And so I thought about structuring groups, assigning groups, and giving people in the groups, roles. So, all of these are just different ways to think about how do I bring more structure to my classroom for all students. And it’s not going to hurt the students that already know how to take notes. And it’s not going to hurt students who know how to take notes on outlines, and all of that, but for the ones that need that, it’s going to really level the playing field for them.

Viji: Yeah. And I’d add to the idea that the technology can help us here. We have a lot of good platforms, not a single one that would do everything, but we have good platforms that help us accomplish these goals. And I’ll give the classroom response system, or polling, the example that I use… that’s something that’s something that I was using, even before I redesigned the course… and the reason I loved it so much was because I could hear from every student in a classroom, right? I didn’t have to wonder if it was just the brave one who raised their hand who understood it and looked and scanned and tried to make sense of the confusion of the faces, right? There’s no ambiguity. If I know that 97% of the people got the question right, then we can move on. That’s a pretty good response. So, thinking through what technology exists to help us help all students is really important in this work.

Kelly: I’m currently really enjoying… in our learning management system,there’s something called lesson tools. And it’s a way to build each lesson for students. And it’s such an easy way to think about building something before, during and after. And I feel like a lot of people are starting to realize that building an online class just requires so much more structure that, as that translates back into the face-to-face classroom, that structure will be built. Yes, it takes a lot of time and effort to build it, but once it’s there, you’ve got all these online homeworks and resources and videos. We’re going to have a lot more ways to say to students, you can learn this this way, or this way, or this way. And that is the basis of universal design, something I think we should all strive to do. But, we know it takes time and effort to get all those resources together.

Rebecca: These are a lot of things that are very dear to my heart too… really thinking about flexibility and making sure that we can engage students in a lot of different ways.

Viji: There are many things about this emergency transition, the change to remote instruction that I think we’re all learning that that flexibility, and the structure, is really important. And sometimes people think that they are at odds with one another, but they’re really not… that we need to think about multiple ways to have assignments be late, for example, because things are happening in life. I think for far too long. we’ve ignored the differences that our students come to the classroom with, and now it’s in our face when we see that a student doesn’t have a good internet connection, for example. So, those differences are becoming very clear in this transition. And, like Kelly, I’m optimistic that many of the things we’re designing and learning will stick beyond this transition, because we are building things that will last… hopefully they’ll last in the courses… the notes you make, the videos you make, these are all things that can be helpful to students in the future as well.

John: That was something we emphasized with our workshops for helping people prepare for the fall back at the beginning of the summer, telling them that “Yeah, this is going to be a lot more work preparing your course then many of you have ever done before, but the people who already were teaching online really didn’t have many problems because they had a lot of the things built. And if you do this, even if this pandemic is gone in a year or so, everything you’ve created can still be used as long as you create them in ways that are modular and that can be adopted for continued use in the future. I think that helped convince a lot of people that it was a good time to start devoting to those activities, because it wasn’t just for a one or two semester emergency, but it was going to be a change that could actually improve their classes indefinitely. At least, that’s what we tried to convince people… there are a lot of really panicked and worried people.

Viji: It’s an investment. It’s a heavy investment, in a short amount of time, in a very panicked way. And we’re sympathetic to my colleagues who are doing this while also caregiving and that there’s a lot… it’s not just life as normal, that we’re asking a lot of a lot of people in a short amount of time.

Kelly: And I like your use of the word modular because for me, that’s really key. I build everything by lesson objective. So, it might only take me 10 minutes to make a video, so I can pop in and out of my life, I don’t have to worry about creating this awesome video with no outtakes, right? …it’s just much quicker. And then students can also say, “Okay, I see I have six videos to watch today, but they’re all five or 10 minutes, I’ll do three now, I’ll do three later. So, I do think it fits nicely with the time we’re in, but it also helps alignment across the course, too, for students to know exactly what they need to do, and then use those modules as the basis for your assessment.

Rebecca: I agree, Kelly, I’ve been spending a lot of time making sure that the modules that I’m creating can actually act as standalone things and don’t connect [LAUGHTER] between them, so that I could mix and match them in the future, because there’s some things that, in a virtual environment, I’m doing in an order that I might do differently if we were in person. And so, I think that’s ending up working really well. I’m having to articulate what I want to articulate really concretely about a particular subject and break it down into smaller pieces. And I think you’re right that that structure is going to stick later on. I’m going to keep doing that in the future and it’s definitely causing me to think about things differently. We’ve talked a bit about the structure of classes and ways that we can be more equitable and inclusive. But what about the way that we evaluate student work and grade student work?

Kelly: One thing that we often talk about in the workshops we do at a lot of institutions is we think about the growth mindset. And the idea that it takes practice to get good at something. And we like to share with students that it takes practice for us and mistakes are part of learning and we hope all educators buy into that. But then when you ask educators, where in your syllabus in your grading policies is the growth mindset. We’ve seen so many faculty just scratch their heads and say like, “You’re right.” This is a philosophy I believe in, but it’s not built into what I actually do. Because we have hard deadlines. We count everything a zero if it’s not there. And so, Viji and I have some ways that we’ve done it, and we’re always trying to think how much more can we put into our grading and our policies that really account for that growth mindset. So, for me, an example is I allow students to drop their lowest exam. And with first-year students in a STEM course, many of them don’t do well on their first exam. And it helped me think about, “Oh, let me give them an earlier failure. Let me give them a hard quiz earlier on so it doesn’t hurt them a lot.” But, allowing them to drop an exam gives them the sense that “Okay, I didn’t do well, but I don’t have to leave the major.” And honestly, students think that… they get one low grade, and they think they’re done with that entire discipline. So, that’s one way I’ve dealt with that growth mindset.

Viji: Yeah. And that point that Kelly made about leaving the major… to some faculty, that might sound ridiculous, like we’ve certainly been knocked down a few times and picked ourselves up. But, there are some students for whom they’ve been told their whole lives, they don’t fit. And if you get that early piece of feedback that, indeed, you don’t fit, and that’s the way they interpret it. It doesn’t mean that that’s what’s actually happening. What’s actually happening is they’ve made a mistake in terms of their preparation, or maybe they didn’t have the right types of study strategies, whatever it is, but we want to convey in our courses that you can recover from that early mistake by using the right approaches. Let’s sit together and talk about what you did do and what we might do better next time around. And so having this grading structure where you drop a grade… In my course, I have a cumulative final in statistics… it’s easy to have a cumulative final, everything sort of builds on one another in terms of content. And I say that if you do better on the final, it can replace one of the earlier exam grades. So again, it builds that opportunity for being able to understand the material at some point, it’s okay, if you don’t get it by the exam date one or exam date two, we’ll get there and it’s not a race. It’s not about getting there at a certain time. It might not even happen this semester, it might take several semesters of chipping away at a certain topic, but that you give them a little bit of grace in terms of the timeline with which they might understand that material. And then again, like does it really have to be a zero if you don’t turn something in versus a 60 or 70 or 80? Right? The mathematical average of that is terrible. So, let’s think about ways in which we can assign grading such that a single late assignment doesn’t harm you greatly or a single low grade doesn’t harm you greatly and bake that into the grading scheme of our courses.

Kelly: And on a bigger scale, when we say we look out into the conferences of our disciplines, or we ask where’s the diversity in our own disciplines, it comes back to these little decisions. This is anti-racist teaching, when you think about these things. By having really hard first exams, that’s a barrier that excludes people, and if we really want diversity in our disciplines, these are the little decisions that we make that are really powerful in terms of the effect and impact they have on students.

Viji: Yeah, we’ve all heard that “Look to your left, look to your right. Some of you will not make it” and then we say as educators “Well, that’s terrible. Why would somebody say that?” But, then you look at our syllabus construction, and really, it’s just a different version of that kind of statement.

John: And I think another thing you advocate is keeping most of your assessments low stake so that way any one thing they may not do well on… besides dropping the lowest grade from a set, just keeping pretty much everything low stakes could also take some of the pressure off and reduce some of that effect.

Kelly: Yeah, absolutely. That’s another great strategy.

John: What are some of the things that faculty do in class that makes class discussions less inclusive? And what can we do to make these discussions more inclusive?

Kelly: Now this is a question near and dear to our heart because Viji and I are often at meetings together and either quietly texting each other or giving each other a look. And we know each other well enough to give a look and know exactly what it means. And a lot of meetings we’re in are just not inclusive. If you’re not the person that’s just going to raise your hand and say something potentially controversial in a room full of ranks and hierarchy. Our students feel that way too. Whether it’s actually ranks and hierarchy, there are lots of reasons why a student doesn’t feel comfortable speaking up. And so a great way to do this is to take the volunteer aspect out of it in a large classroom and put them into smaller cohorts. And many students are very comfortable talking to each other in small groups, verifying their ideas, building their confidence that what they are thinking has merit, is a great way to start building community and to have students start feeling comfortable. And once they’ve gotten that affirmation in a small group, more people are willing to represent what their group said. So like, for instance, I never call on an individual student… cold call and say, “What do you think?” I always give them a chance to talk first. And then I say, “Okay, group number 63, it looks like your numbers up, what is your group talking about? Fill me in.” And so I’m hearing a diversity of voices, but I’m also trying to make the environment a safe place where people can build their own communities as well as contribute to the larger community.

John: And people would feel more comfortable when they’re representing the group discussion than presenting their own. So that takes a lot of the pressure off, I think,

Kelly: Yeah.

Viji: Yeah, no one wants to be wrong, and especially in front of the professor and their peers, right? So, they’re simply reporting for the group and that’s the group’s discussion. And as skilled educators we all know how to turn a wrong answer into a learning opportunity in a classroom, but it still doesn’t take the sting away for that person who feels like they may not speak up again because of it. So, anything we can do to make it feel comfortable to be incorrect, because it’s still a learning opportunity or to say, “Well, that’s a common misperception. Let’s break that down a little bit and talk about it some more.” Those kinds of things really go a long way to building the confidence of the student. I remember one student, in particular, who wrote me just such a kind note at the end of the semester talking about how this is a common refrain in my course… they have not been looking forward to taking a statistics class… Shockingly, there’s not a lot of people who say that they are looking forward to it… But, in this case, she wrote to say, beyond any sort of content lessons I provided, what I provided to her was the opportunity to understand that she was right a lot of times in her group discussions, even though her peers tried to convince her she was wrong. And she began to doubt herself. And she’d pull in her answer because the group had a different answer, and then she realized originally she was right. So, she built confidence, but she also learned that she really knew what she was doing and she didn’t understand that about herself and she had more conviction after she left that course to be more forthright about her opinions in other settings. So, these are the kinds of things we can do when we add structure for giving people a chance to reflect on who they are as a learner and who they are as a person and how they can contribute in their groups and in society.

Kelly: I’d also like to add that we don’t have to have people speak up to be part of a community, that there are lots of other ways to contribute and writing, and using anonymous polling systems, these are all such great tools, and they’re the ones I certainly would have gravitated to as a student, had I ever been given the opportunity. I spent four years as an engaged high achieving student in college and never once raised my hand to participate, it just wasn’t what I was going to do.

John: Yeah, and polling gives people the same instant feedback, so they know whether they were right or wrong, but from a class’ perspective, it feels anonymous, that they’re not putting themselves out there where they risk the embarrassment of appearing to be wrong.

Rebecca: One of the things that I have certainly seen a lot of conversation about currently on Twitter, and I know that you’ve both engaged in these conversations about, is how to community build at the beginning of a class, especially in virtual environment where you have that really awkward online silence, and nobody really knows what to do with. [LAUGHTER] And you’ve offered some interesting ideas, would you mind sharing some of those?

Viji: When we are used to teaching in a classroom space like, in the same building together, I hesitate to say in person, because we’re still in person in this environment. But, when we’re together in a classroom, there’s a buzz that is at the beginning of the class time, right? …so that people are chatting with their neighbors; it feels like a warm environment, oftentimes, when you walk into it, at least the classroom where the conditions are right. You feel a warmth when you come in, that you’re going to be learning, and when you’re online, it’s really hard to simulate that kind of buzz because of the nature of the tool. So, thinking about ways you can have that kind of chitchat is really helpful. So, I use polling in this environment, as well, right? I can have a question posed on the screen and students can respond to that question either in the chat window or through Poll Everywhere. I like using Poll Everywhere because I use it anyway. The downside to using chat in some platforms is if you join late, you don’t see the previous responses. So, if you could use something where students can scroll through and see their peers responses, that’s a nice way to kind of get warmed up for the class session. It might be something about, you know, what they’re grateful for today. Or maybe they could tell you a little bit about something that they ate recently that they really enjoyed. But, just getting some small talk in before having something in place that gives a little structure. I’ve heard people talk about playing music, just any small ways you can to try to bring some sense of community in those moments before class start, I think is really helpful.

Kelly: And I would agree, Viji started teaching in the spring online with some synchronous sessions. I was doing asynchronous, so she told me to do it. I did it, and it works. It’s a nice anonymous way to have that chit chat too without owning it in the chat box. I’ve used it selfishly this semester already to find out how students are doing, if there’s something I could do better for them, just taking the pulse. So, a bit of a survey question as well. My daughter is in high school. She just started high school and, of course, it’s online high school. And I keep asking her, “Did you get into your session on time?” And she goes, “No.” And I said, “Why? Why not? [LAUGHTER]She goes, “Well, I want to be a little bit late.”

“Why?”

“I don’t want to be the first one there.” She’s so afraid of like, how awkward it is that she can see on the platform there on how many people are there. [LAUGHTER] And at some number, that’s when she jumps in.

John: As long as everyone doesn’t do that, then we’d have a bit of a coordination failure. [LAUGHTER]

Rebecca: I don’t know, as a faculty member, I don’t want to be the first one there in an awkward silence either. [LAUGHTER]

Viji: But, that’s just the point. It doesn’t have to be awkward. Why not just design it so it’s less awkward? We all know it. We all go into these things. And we’re like, “Oh, it’s another one of those starts to the meeting,” right? But let’s just make it so that we have something that we respond to, that we see on the screen, everybody can see it. It’s also awkward, I think, when you walk into a meeting, and they’ve started, and they’re talking about something, but you have no idea what they’re talking about or how to jump into that conversation. So, having a prompt on the screen is one way where everyone, even those who come late, can still see what the conversation is about.

Rebecca: I’ve had a couple of colleagues who are also using whiteboard features in video conferencing software to have like a doodle board where people can collaborate or Doodle… we teach art classes… doodle on the board, and collaborate as a way to silently do something together. That seems to be pretty effective as well.

Viji: Yeah, I love that idea.

Kelly: That’s a great idea. I’m gonna do that next time. Thank you.

John: In the chapter that you wrote for Teaching Statistics and Quantitative Methods in the 21st Century, you mentioned using polling tools to provide challenging questions to students. Do you do that in a single-stage process? Or do you have students vote first and then discuss it in smaller groups or with pairs before voting again?

Viji: That’s a great question. A lot of it has been through trial and error, understanding what was a hard question and then breaking it down to something that’s a little bit simpler. So, if it’s a multi-step problem, I’ve learned to scaffold the problem through multiple polls and then get them to the right answer. It’s very helpful in quantitative work because people do work at different pace. And so this can level that playing field by getting everybody at the same stage of the problem through the scaffolded polls. But, there are some polls that I know really work very well as a “Give me your thoughts first.” And then let’s do it now where we talk to one another, we do a bit of peer instruction, and then we re-poll. And I love showing them the results from round one to round two… I call them round…, because then I say to them, “See, you don’t actually need me here.” But the truth is they do. They need me to pose the question, they need to get in there and tease out the problem that I know that they’re going to have challenge with, but they can do the work of teaching each other the material and getting through the problem together, and on the whole getting it right. So, those are fun ones for me, because it’s also about building community and they love it. They know that like my goal for every poll is that 100% of them get it right. And so that’s another way I convey that it’s important to me that all students learn the material

John: If we’re teaching remotely, synchronously what can we do besides meeting with them at the beginning of class and just chatting with them and maybe at the end of class, what else can we do to make that environment more inclusive?

Kelly: Well, one of the things I love about this environment is everybody’s name is up on the screen, which helps me a lot as an instructor, but it helps them know each other, too. So, it can be community building. And it’s a great way for people who have names that are difficult to pronounce to put a phonetic spelling, to ask people if they would like to add a pronoun there. I think these are advantages that we just haven’t figured out quite as easily in the face-to-face classroom. I use note cards in my class for the same reason. But, I can’t tell you how many times they either refuse to take them out or forget them. So, it’s never the hundred percent I get on a Zoom screen with names. But, one thing I’ve noticed people talk about often is the back channel. So, having the chat going, and it seems to be universal that people are feeling already a little bit sad about when we lose chat, when we go back to the face to face or in the same room environment that there’s a lot of good discussion that happens in that backchannel. And I know people do use backchannels in classroom spaces too. That’s one aspect of this environment that’s unique and helps bring more voices to the table. I think another thing that is worth mentioning is, I would hope people are using their live sessions for doing those difficult things together and not talking at students because that could be better served with a video. I’m sure we all find ourselves explaining and talking at times. So, I think one thing we could do is to help our students is to say you don’t need your camera on right now, although I’d love to see you and it helps us build community, this could be a time when you could turn your camera off. I also have invited my students to use virtual backgrounds, because when I’m teaching, I’m in my bedroom, and I think it’s odd to see your professors’ bedroom, so I use one, but I think it’s a nice talking point too. If students feel more comfortable, if they are going to share their camera, then maybe they don’t want to share their surroundings. So, just not just assuming students all know that, to be very explicit and say to students, “Here are all the different ways that you can access this course. You don’t have to turn your camera on, but here are the ways that I think I would love to see you engage.”

Rebecca: You’ve both written a bit about the hidden agenda, or the hidden curriculum, of using these kinds of tools and technologies, and you have a Student Guide for using Zoom. And I took all of that to heart too, and made sure that I made some videos about the different kinds of tools we were using this semester, and actually built in the whole first week of just like, this is how we do the things. And like, let’s try them. [LAUGHTER] And then there were some ways that I was planning on using some tools, and we’ve actually already pivoted, because it didn’t quite work the way that I had hoped. And now we have something that’s working a little bit better for everyone. So, I think that’s also an important piece to point out. Can you point out some of the features maybe of the guide that you created for students?

Viji: Yeah, I mean, what you’re talking about is what we’ve been talking, about adding structure to these tools, right? So, just because it’s in front of them doesn’t mean that they know how to use it. We all saw a car before we knew how to drive, that doesn’t mean we knew how to drive it. And everybody thinks it’s very intuitive, but again, what do you do when you start a meeting? Do you turn your camera on or off? Do you mute… on or off? What does it look like to say goodbye in a Zoom meeting? There are certain things like that, that I, at least when, as Kelly mentioned, I switched to synchronously meeting because we were doing all these problem-solving sessions. I wanted to keep that as what our synchronous meetings were. And I was anticipating that some students would have questions like that. This actually started with somebody tweeting about having a dress code for showing up to a Zoom session, and I just thought, are you kidding? There’s a pandemic going on, and you’re thinking about what the student is going to wear to come to class, when they’ve been moved out of their dorm, sent home, barely have internet, there were so many things where I thought I just need to let them know that that is not on my mind. I don’t care. I’m just grateful that you’re alive and you’re continuing to learn. So, those are examples of things that I wanted to think through and Kelly helped me think through like, “What kind of questions will come up?” …and we brainstormed ways that we could just communicate it in ways that students, hopefully they find them to be just the synched answers to questions that they might be wondering and not sure how to ask or if it’s appropriate to ask and what to wear was one of those things.

Kelly: And that’s a good example of the shared brain we have some times, because I called Viji one night and I said, “You know, we should write something up about being more inclusive with Zoom.” And she goes, “I was just writing a guide for my students.” And so we just quickly put it together and had a lot of the same ideas around that. Coming back to the idea of the hidden curriculum, I think that same idea where a lot of us are new to using Zoom and these different tools, that we remember how hard it was to get on and what the rules of it were. And they’re constantly changing, the settings and all of that. So it might seem obvious to make a guide for your students about how to use Zoom. But, what are the other aspects in our teaching that we take for granted? We’re such experts, and we’re so comfortable with the college classroom, I think we always have to be asking ourselves. “What other guides should we be writing that seems so obvious to us?” We forget that we’ve been here a long time and we don’t want students to feel like there’s this culture they don’t know about.

John: I actually put a note in my syllabus telling students that while they’re invited to use their cameras, they’re not required to. and if they’d prefer, they could put up a picture of themselves or of their pet or of anything that they’d like to use as a symbol for that day, because it probably would look nicer to see images of people than those just little black boxes on the screen. And they responded pretty positively to that. I did send out a note to our faculty before classes started this semester suggesting that faculty should invite students to use cameras ift they felt comfortable, but should not require it. And the response was not quite as positive. A lot of faculty seem to believe that they need to see their students to make sure that they’re there, to make sure that they’re engaged, and to look into their eyes to measure whether they’re learning, [LAUGHTER] because apparently their eyes provide secret signals to some faculty about the amount of learning that’s taking place. It generated a lot of emails,

Viji: They have some tools that I don’t even know about. I didn’t know there was such a tool that I could use,

John: it does suggest perhaps the need for more inclusivity training for faculty.

Rebecca: I had one last question about Zoom environments and things and that’s about microaggressions. We know that we need to shut them down when they occur, but I think that faculty, if they’re not used to being in a virtual environment, whether an asynchronous online chat or discussion board, or in a Zoom session, figuring out ways of handling situations just seems different. Do you have any advice for how to handle those kinds of situations in those different types of environments?

Kelly: Well, I think you hit on it already. One thing that’s common in all of these environments is don’t ignore them. Right? If it’s asynchronous, then like, say something was put on a discussion board. I personally would feel like ‘Oh, phew, I have a minute to think about this without everybody staring at me.” Right? And so each case is going to be a little bit different in terms of how you deal with it. You also can’t pull aside the people after class who may have been impacted by that. So, we have to remember, whatever we do to deal with it, should also include really reaching out and being mindful of who those students are that might have been impacted. I would say live online is probably not that different from in a classroom. because we have to do something at that moment. And that could be saying like, “Let’s take a pause, let’s stop.” My instincts and teaching are always to turn it into a teachable moment and to turn it back on them and say like, “This is what just happened. Can we all take a moment to maybe reflect? to put into writing the impact this could have on a student?” You know, something where I personally just need a moment to think, and I’m not going to be embarrassed about that, and I think that my students will come up with a lot of things I wouldn’t have come up with in a very eloquent way of dealing with it.

Viji: Yeah. And I think the only thing I’d add to that is it feels scarier in this online environment, because oftentimes, we are recording sessions. People can snapshot even though we might set good intentions with our students about what they can and can’t share with an outside community, we can’t control it entirely. And so it can feel even scarier, I think, to feel like there’s some level of posterity around that moment or your reaction to that moment. So, I think, if anything, I mean, we’ve had a lot of discussions in the world about different kinds of discrimination and all aspects of life that are harder for some students… not ignoring it is definitely the first step. I think there’s even the step before that, which is, I might not recognize it. So, how can I support you as learners. And as peers, if you see something, I’d like to know what it is, even if I am the one who’s doing it. I want to know because I want to do better. So, really being open to that kind of criticism from students or just acknowledging that you’re a human being like all other human beings, and you’ll make mistakes and inviting them to help you become a better person by suggesting that this is going to happen. It’s inevitable that something like this will happen, but we should be models of how to deal with that situation and be productive in our conversations about it and to move forward on it, right? We don’t want to shame anybody for doing something that might not have been their intent, but the impact is no less to the people who have experienced that microaggression. So, really thinking through and planning for it happening and talking about what you’d like to do as a community of learners. But yeah, as Kelly mentioned, if it’s asynchronous, you’ve got a moment, you can gather yourself, you could talk to your peers and say, “Hey, this happened, what do you think is the best approach?” But, if it’s not asynchronous, I think it’s fine to just say, “Hey, let’s hang on a second, I need a moment to just think about what happened here, and how we might respond to it.” And it might be, we might need to come back to this at the next class session, and give yourself that time to think through it. But, I think even the students who may have felt slighted by it will appreciate that you hit pause for a second, and you’re willing to work through it and that you trust them to make the right decisions moving forward to learn from it. And I think going on what Viji said about maybe a little bit of prevention, some practical ways you can invite that feedback in an anonymous way is to use a Google form that is always open. You can set it up so that you get an email if there’s something there and students can report on anything relative to the class, but especially microaggressions that you may have performed without knowing or classmates, if they’re doing group work, you certainly can’t monitor everything, you’re not in all of those spaces. And then coming all the way back to setting up group contracts and respect and civility in whatever kind of mode and classroom you have that semester. Hopefully, you get to a place where you’re preventing some of these things, but also recognizing that they will happen.

John: You both have a book coming out from West Virginia University Press. Could you tell us a little bit about what the book will be about and when will it be available?

Kelly: Well, the book is definitely about inclusive teaching. And spoiler alert, it is definitely about structure. [LAUGHTER] And we really walk through course design, facilitation, but we’re also really thinking about all aspects of a course. So, whether it be office hours or communicating with students or bringing in undergraduate learning assistants, whatever parts of a course that enhance learning, we really want to think about structure in all of those areas.

Viji: Yeah, and one of the challenges we faced is we’ve both read a lot about good teaching, right? So, a lot of these practices are good teaching, but we wanted to apply the lens of how it promotes inclusive teaching through this book, so that, ideally, the reader would then be able to take some of these themes and see them and apply them in other areas that we didn’t explicitly talk about. So, just a way to view the world as you’re teaching and thinking about how to add more structure, and the idea that if we leave things to chance that some students will be left behind, and that’s really not acceptable.

Kelly: As far as the timeline, we’re not sure. Our first draft is in, snd that’s all we can say.

John: Excellent. So. that’s a fair amount of progress, because you just signed it not too long ago, if I remember seeing it on Twitter.

Kelly: Yeah, it was fun to write together. We definitely get in a groove with writing some sentences together. And then sometimes it was just you write this, I’ll write this, and we’ll swap. But, it’s certainly a way of knowing someone pretty deeply when you write a lot together.

Viji: Yeah. And we often talk about the benefits of diversity, right? And so doing these projects of writing, but also, when we do our workshops, we speak a lot. And when we come up with ideas about what we might do, it’s always great to be able to bounce ideas off of each other and to say, “But what if we tried this” and “we did this” and well, you get that second person really reflecting on some of the ideas, and it’s really helpful to be able to do that and you get a better product, quite frankly. No matter what it is, it’s better when more people can critique it and give you feedback about it.

Rebecca: And we’re all going to benefit from that collaboration because we’re all looking forward to your book.

Kelly: Thank you.

Viji: Thank you.

Rebecca: So, we always wrap up by asking what’s next? So, we teased you [LAUGHTER] You already said about your books. Now you have to come up with something else.

Viji: You mean what’s my next beverage after I finished this LaCroix, or…

John: It could be.

Rebecca: It could be whatever, yeah… I’m gonna go take a nap, whatever it is…

Viji: Well, literally what’s next is I’m going to get out of my seat because I’ve been in it for a long time and I’m probably going to take a walk with my son who’s home, this his home day. He is learning from home today, and then I’m sure I’ll sit back down at the computer and answer some emails and, I feel like these days, it’s one day at a time and eventually I’ll get to the point where I can look a few months ahead. But, for right now, it’s one day at a time.

Kelly: For me, I guess I’ll take a much broader view, and an optimistic point of view, that I think what’s next is, once we get through this crisis, that teaching and the way we educate our students, I think, is going to come out better for what we’ve been through, because I see people doing the best they can in this environment, but really paying attention to how learning works. And I think our students will be winners in the long run in that, however we come out of this.

John: Thank you. It’s wonderful talking to you. Thank you for all the work you’ve been doing in supporting instructors all over the world for quite a while now. We’ve appreciated it and we share a lot of the things that you’ve done with our faculty.

Rebecca: Yeah, thank you so much. It was really wonderful hearing from all of you.

Viji: Thank you.

Kelly: Thank you. Thanks for having us.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

148. Active Learning: 6 Feet of Separation

Posted in active learning, clickers, edtech, evidence-based practices, growth mindset, higher education, hybrid, hypothesis, learning, online learning, pedagogy, peer instruction, remote learning, social presence, teaching
August 12, 2020

During the fall 2020 semester, many faculty will be working in a classroom environment in which they will be in a classroom using a video conferencing tool to work simultaneously with a mix of remote students online and masked and physically distanced face-to-face students. There are significant challenges in using active learning techniques in this environment. In this episode, Dr. Derek Bruff joins us to explore some active learning strategies that may work under these very unusual circumstances.

Derek is the Director of the Vanderbilt University Center for Teaching and a Principal Senior Lecturer in the Vanderbilt Department of Mathematics. He is the author of Teaching with Classroom Response Systems: Creating Active Learning Environments, as well as his most recent book on Intentional Tech: Principles to Guide the Use of Educational Technology in College Teaching. Derek is also a host of the Leading Lines podcast.

Show Notes

Transcript

John: During the fall 2020 semester, many faculty will be working in a classroom environment in which they will be in a classroom using a video conferencing tool to work simultaneously with a mix of remote students online and masked and physically distanced face-to-face students. There are significant challenges in using active learning techniques in this environment. In this episode, we explore some active learning strategies that may work under these very unusual circumstances.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and

Rebecca: , a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guest today is Dr. Derek Bruff. Derek is the Director of the Vanderbilt University Center for Teaching and a Principal Senior Lecturer in the Vanderbilt Department of Mathematics. He is the author of Teaching with Classroom Response Systems: Creating Active Learning Environments, as well as his most recent book on Intentional Tech: Principles to Guide the Use of Educational Technology in College Teaching, which we talked about in an earlier podcast. Derek is also a host of the Leading Lines podcast. Welcome back, Derek.

Derek: Thanks. I’m glad to be here. I’m glad to be back on the podcast. And I just want to say I’ve been very impressed at the work you two have been doing these past several months. I used to say Leading Lines comes out on the first and third Monday of every month. And now I say Leading Lines comes out when it comes out. [LAUGHTER] And so, keeping up the schedule that you guys have had with this podcast and bringing in so many great guests and having so many great interviews, it’s just been a really rich resource for me. And as someone who can’t keep up a regular podcasting schedule right now, I’m just very impressed at what you guys have been doing.

Rebecca: It’s all John.

John: We’ve gotten a lot of help from so many people, such as you, who have agreed to join us and share their thoughts in a really challenging time. And it’s been a really great resource for our faculty too, who are faced with all this uncertainty about the fall.

Rebecca: Me too, because I’m not teaching until the fall. I haven’t taught this spring. [LAUGHTER]

Derek: Now’s a good time to be on sabbatical.

Rebecca: So, today’s teas are. Are you drinking tea, Derek?

Derek: No, I have some dark roast coffee.

Rebecca: Caffeine. [LAUGHTER]

Derek: Yeah.

John: And I have ginger peach green tea today.

Rebecca: I have a summer berry green tea. See, I’m mixing it up, John.

John: That’s a new one.

Rebecca: I gotcha. I gotcha.

Derek: Yeah, sounds lovely.

Rebecca: Actually, you’d be very happy to know, John, that last time I was in Epcot. I got it. [LAUGHTER]

John: We’re recording this in July. It’ll be released probably in early August. And there’s a lot of uncertainty about the fall. Right now, probably most colleges and universities in the country, with a few notable exceptions, have announced that they’re planning to bring students back to campus for face-to-face instruction with reduced seating, with some students coming in remotely (typically through Zoom or some other video conference app), and you recently released a blog post that discuss options for maintaining active learning in this environment where some students will be there in the classroom, spread out to make it hard for them to be in contact with each other, as well as online with a video conference. Maybe you can talk a little bit about that sort of framework in terms of what possibilities there are for people to interact.

Derek: Sure. I wrote this blog post because I’ve been getting so many questions from my faculty at Vanderbilt, trying to imagine what the fall semester will be like. And I gave a presentation based on the blog post at a Vanderbilt faculty town hall the other week, and one of my slides said, “No one has ever done this before.” And I think that’s really important. So, what I’m imagining that a number of faculty will face this fall, and again, different campuses are making different decisions… and even within a single campus, there’s going to be a lot of different configurations… some faculty will be teaching fully online, some faculty will not… and so this is kind of what we’re trying to imagine is that if I’m teaching a class this fall and I’m back on campus, I’m in a classroom, I have some students there in the room with me, but because of social distancing requirements, they’re six feet apart from each other. Maybe we’re all wearing masks, maybe I have a face shield instead of a mask. There’s going to be some variability here. But some students will not be able to come to the classroom, either because the social distancing requirements mean that you can’t hold as many students in the classroom, or maybe they are unable to travel back to get on campus. A lot of our international students are not planning to come back on campus this fall for a variety of reasons. Or maybe you have a student who’s in quarantine, right? They’ve been exposed to COVID and they’re in quarantine for 14 days, and so they can’t come to the classroom. A lot of us teaching this fall. I don’t know how many, but a lot of us, are likely to have these classes where some of the students are in the classroom, but physically distanced and masked, and some students are participating at the same time, but virtually, perhaps through video conferencing tool like Zoom. And you may have some students who actually can’t do either of those. If they’re 12 time zones away, they may have to participate in your class asynchronously in some fashion. That’s a whole ‘nother level of challenge. In my blog post, I just focused on those first two groups of students, the in-person but physically distanced, and then the virtual students may be participating via Zoom. And I’ve been really careful in my terminology of how I describe this, and so I’m calling this a hybrid classroom, because hybrid is sufficiently generic [LAUGHTER] that it would apply to a situation like this. Hybrid typically means some combination of face-to-face and online activities. It’s not quite hyflex. So there’s this term you may have heard, hyflex, which I think means something more specific, where students really have a choice to participate in person, synchronously online, asynchronously online, and they may actually shift from those modalities over the length of the semester. The flex in Hyflex is the kind of student choice and the student autonomy piece and I’m not expecting we’re gonna have a lot of student autonomy this fall. We have some, certainly. Students are electing to be remote-only students or on-campus students, but it doesn’t feel like it’s quite hyflex in terms of the classic model there. That said, though, the folks who teach in a hyflex environment have developed teaching strategies that can work when you have students participating in the classroom and students participating online at the same time. That said, they haven’t had to deal with the physical distancing and the masks. That’s the part that really is novel, and I think it’s going to be important that we as instructors give ourselves a little bit of grace, knowing that literally no one has taught under these conditions in the past, this is new for us, it’s new for our colleagues at other institutions, it’s new for our students. This is going to be really weird and really challenging for them. And frankly, there are a lot who would argue that it might be easier to just teach fully online; that trying to kind of juggle the constraints in this kind of classroom is going to be really challenging for a lot of faculty. And I’ve talked to a lot of faculty who are like, I would just rather teach fully online this fall. And so I don’t want to speak too much into that choice. There’s a lot of factors that go into university decisions about kind of bringing folks back to campus this fall. Our work at the Center for Teaching, we’re trying to help faculty teach as well as they can in whatever conditions they find themselves. We don’t usually get to pick those conditions, right? And so, I just wanted to try to be helpful and so I wrote the blog posts because using technology to foster active learning in the classroom, that’s my jam. That’s what I’ve been writing about and speaking about for a decade and a half now. I wrote my book on teaching with classroom response systems back in 2009…. clickers and polling software… we have tools, actually. This classroom setup sounds really hard, and it will be hard. And there’s stuff about it that I can’t predict in terms of how hard it will be. But we do have some tools and technologies that can help foster more active learning in these types of environments. And so that’s what I wanted to lay out for my faculty colleagues, who couldn’t imagine how this could work at all. And I could see a few ways actually… a few tools that could make it functional. And so I wanted to share those strategies.

John: Before we talk about the specific strategies, maybe we could talk about some of the mechanics. One of the things you suggest is that you’re assuming that the people who are in the classroom will be able to see and hear the people speaking from outside over Zoom or some other videoconference tool, but what about voice going from the classroom to remote participants. Since the in-class students won’t be able to use their own microphones (to avoid possible feedback effects), how will students who are participating remotely be able to hear what other students say in the classroom?

Derek: Right? And that’s where I don’t know that there’s going to be a good answer to that. The audio piece is one of the more challenging pieces of this classroom environment… the students in the classroom getting to hear each other when they’re sitting far apart and wearing masks… but, even more so, the students who are participating virtually. How can they hear the students in the classroom? Presumably, the instructor will be running Zoom off of the classroom computer or their laptop, and they’ll have a webcam and a microphone. If I’m close to my computer when I’m running Zoom, people on the Zoom call can hear me. So, having the virtual students hear the instructor seems fairly straightforward, but the students in the classroom, they’re not going to come anywhere near that microphone. Now we have a few classrooms at Vanderbilt that have some ceiling mounted microphones that are going to help with that, and that will have some capability to pick up the student voices in the room. But, I’m anticipating that’s going to be a real challenge, actually. And so, again, I don’t have a silver bullet, but it does speak to the use of something like a backchannel tool. So like a text chat in a Zoom room or using a third party tool like Twitter, or GroupMe, or Slack, or Discord… a place where you can have a text conversation with all of your students during class. This is often called the backchannel. The front channel, it’s kind of you at the front of the room talking and having conversations with your students. But the backchannel is the text chat that kind of supplements that. And I’ve been doing stuff with backchannel for years. It’s a really interesting way to build community in the classroom, to give voices to more students to kind of create an on-the-fly closed captioning almost or documentation of the discussion that’s happening. It can be really powerful to have a good backchannel. It can also be really challenging. I think a lot of instructors who’ve dabbled with this have realized that when they are at the front of the room leading class, they don’t have the bandwidth to also pay attention to the text chat and see what’s happening there. And so what I’ve recommended is what I learned from Steve Gilbert and Steve Ehrmann years ago, doing webinars to have someone called the “voice of the chat.” Designate someone, maybe it’s a teaching assistant if you’ve got one of those, but it could be a student in the room, and it could rotate among students over time. Their job is to pay attention to the text chat, the backchannel, and then you as the instructor every so often, you would pause and turn it over to the voice of the chat and say, “What’s been happening in the chat? What are the questions that are emerging there? What are some ideas or comments that are really valuable?” Maybe the voice of the chat is someone on your Zoom call, one of your virtual students, because everyone will be able to hear that student. But that way the students who are in the room and the students who are participating virtually can engage in conversation during class, but in the text chat, and then you have these moments where you pull that conversation from the back channel to the front channel, using the voice of the chat. I’ve done this in a lot of online activities, and it works really well. [LAUGHTER] When I’m going to do an online webinar of some sort, I want to have a voice of the chat, someone who can play that role. And so this is actually a pretty proven technique. And I think it’s going to be fairly practical for our classes this fall. I’m glad you asked about this because class discussion. as we think of it in the generic sense, may be the hardest thing to do in this hybrid environment. For me to stand at the front of the room and show them how PowerPoint slides and lecture to my students, that’ll be relatively easy. That’s also something that you could do without students in the room, right? If you’re just gonna lecture then maybe that’s a pre-recorded something that you share with your students. That doesn’t have to be a live interaction of some sort. But the class discussion, the kind of student-to-student piece is going to be really challenging. And so back channel is one way to try to foster some of that in the classroom.

John: So, the students in the room would be encouraged to bring a mobile device to participate in the text chat and to avoid the feedback loop that would result if they were participating in the video chat using audio. That sounds like a really effective solution.

Rebecca: I think one of the other things that you had some interesting ideas about too was group work. One of the reasons why being in class could be appealing to someone is the idea of being able to collaborate or work on something together. But again, same problem as discussions.

Derek: Right. Yeah. So here’s where I’ve done this a couple of times, just because it was fun in my regular classes, is to use a Google sheet as a way to structure groups and their work and their reporting out. And so, years ago, in my stats course, we had an infographics project. So they had to do some data visualization. And so to get them ready for that, I had them look at some sample infographics. And I invited them to essentially crowdsource the rubric that we would use for the infographics they created. I set up a Google sheet that had across the top, it was kind of levels of quality from poor, acceptable, good, to excellent, and each row was blank. And the idea was that the students would work in small groups, they would look at these sample infographics and they would start to identify what are the components of a really good infographic, and each group would pick a different row on the Google spreadsheet and start to flesh out that component and how you would assess it from kind of poor to excellent. I think I had 100 students in the room when I was doing this. And they were working in groups of two or three, and there were all of these anonymous aardvarks all over the Google sheet, adding their ideas for the rubric and it was a little chaotic, but what I wanted from them was more than just a polling question, this wasn’t a multiple choice question. I wanted them to produce something. It was free response, but not just free response, it was a kind of structured free response. And so the Google sheet was a really nice tool for doing this. And so this is what I’m imagining doing thi fall. You can do this in a fully online, kind of a Zoom session, class. Or if you’ve got this kind of hybrid situation, imagine giving your students three questions to discuss in their small groups. You set up a Google sheet, put those questions at the top, one in each column, share that Google sheet with your students, and then send them off to do their small group discussion. As students in the room…. hopefully, this is part of the unknown… if you have two or three students in the classroom, six feet apart from each other wearing masks, will they be able to have a useful discussion as a small group? I hope so. Again, that’s part of the reason for being in the same place at the same time is to have that kind of student-to-student interaction. I don’t know that I would try groups of size six this fall, I might do groups of size two or size three. And the idea is, they would work in small groups. Meanwhile, on Zoom, your students are probably in breakout rooms, again in groups of size two or three, and they’re talking about the questions that you’ve given them, and they are reporting out, each group on a different row of the Google spreadsheet. Now this does a few things. One is you can monitor the Google spreadsheet as students are putting their responses in there. And that allows you to get a sense of how fast they’re moving through the work, when they’re starting to wrap up…. Oh, most of the students have answered questions one and two, but they’re really slowing down on question three. This is helpful information. You can also start to preview their responses and see what ideas are they bringing to the conversation, and that’ll set you up for whatever you do after the group work, to have a sense of what they’re saying. But, I’m also imagining, it’s a nice structured way for the groups to report out, to share, maybe even to focus. It may be that in the classroom, it’s hard for students to hear each other a little bit. And so you could even imagine, if we’ve got three people in our group and three questions, that each of us will draft a response to one question, and then we’ll rotate and revise each other’s drafts. And you can do that right there in the Google doc. This will take some creativity, it may take a little more coordination than you’re used to needing to do for in-class group work. But it’s also nice that, in this case, the Google sheet as a reporting structure would be the same structure for both your in-person and your online students. And so that simplifies things a little bit. And so, I can imagine that kind of technique working pretty well. Again, there’s a bit of an unknown about the students in the classroom and how well they’ll be able to hear each other. But, that would enable a form of group work that I think would be pretty functional. And it wouldn’t have to be a Google sheet. There’s lots of other online collaborative tools that you could use to have students report out in some fashion. There’s these kind of Whiteboard apps where you’ve kind of put sticky notes all over the board. So it could be something more like that. Or maybe they create a PowerPoint or a Google slides, each group has a different slide where they’re gonna put their answers, they’re gonna put their deliverable of some sort. Again, this is not maybe how we want to do group work, but I think it would be functional in the settings that we’re looking at this fall.

Rebecca: I think one thing that you mentioned in your article, which I also strongly advocate for is if you’re going to use some group work techniques, if you establish something that’s consistent so that you don’t have so much startup cost every time you do group work, that that might help too, for that consistency, and then you might get better responses I would imagine over the course of the semester when there’s less cost in terms of figuring out how to do the thing.

Derek: Absolutely. The first couple of times you do it, it’ll be awkward and hard and slow. But after your students have done it a few times, then it’ll be a lot easier to just kind of slide into this mode with your students.

John: You mentioned the use of polling. And when we moved to remote instruction, I continued that using Zoom, but we were completely remote. The way I did it, and I think this was something you recommended, something Erik Mazur had done, is you poll students with challenging questions, and then you have them work in small groups. In Zoom, that’s pretty easy. You send them into breakout rooms with groups of two (or maybe three, if you have an odd number of students.) How would you do that same type of thing in a classroom setting where you want people to engage in active discussions? Might that be a little challenging in the physical room where everyone can hear everyone else, given that they’re spread equidistant apart somehow?

Derek: Yeah. And so you know, I can imagine doing a polling question, having all of your students respond using the same polling tool. And again, this would assume that your in-person students have a device with them that they’re going to use to participate. Now it may be hard for them to do that via Zoom. You’d have to make sure everyone in the room had their microphones turned off and their audio muted because otherwise you’d have too much audio feedback. But if the students in the room were also in Zoom, but kind of silenced and muted, they could participate in the Zoom polling questions. Or you could go to another tool like TopHat or Poll Everywhere, something that lives outside of Zoom and do all of your polling there in parallel to your Zoom session. Either way, this does not seem to be the semester where you want to put a laptop ban in your classroom, we’re gonna need those tools. And you guys know, I’ve been advocating for years for effective intentional use of digital devices in the classroom. [LAUGHTER] So, we’re gonna need it, we’re not gonna have a choice. But now, let’s say you want to have your students turn to their neighbor and chat about the question. Again, in a normal classroom situation, that’s one of the easiest things you can do to build some active learning into your classroom. Give them a good hard question, have them answer it via the poll, then turn to their neighbor and talk it out together, see if they can put their heads together and get the right answer, and then maybe do a second round on the poll and see where things have shifted. It’s a great pedagogical structure. In the hybrid classroom, the turn to your neighbor and chat is going to be challenging, and so you could try to send your online students to breakout rooms and have them talk there and your in-person students pair up and talk to each other six feet apart. Again, until we do some more testing, I don’t know how practical that’s going to be in the classroom, I’m hopeful that it’ll kind of work. It may be that, what in a normal classroom, you might have them turn to their neighbor and talk for 60 seconds, and then move on, that may be too hard to do. And so if you’re going to have them do group work, you’re going to have them spend 10 minutes doing group work because they’re moving to a Google sheet or they’re doing something kind of bigger and more structured. The kind of quick informal pair work may be too challenging. One option that someone suggested to me that I thought was kind of interesting, though, was to have your students in the room, if you have paired them up with your virtual students, you can have the student in the room pull out their phone, put in their earbuds, and FaceTime with their virtual student partner to talk about the question. Again, the first time you do it, there’s a matching problem there, there’s logistics, there’s audio to figure out. The third or fourth time you do it, this may be a lot more fluid and an easier way to have students chat about the question at hand. It also has the added benefit of connecting your virtual students with your in-person students in more intentional ways. And so that could be really helpful for social presence and things like that. Again, a lot of this is going to be trial and error this fall and figuring out what works and what doesn’t work.

John: Since you won’t necessarily have the same number of students online or remote, and it may be difficult to do that pairing, the pairing could work no matter where the other student was. If you’re in a lecture hall that seats 400, and you’ve got 100 people there or 80 people there, they might call someone 100 feet away… 200 feet away…

Derek: Sure. [LAUGHTER]

John: …which could work in the same way as if they were calling someone remotely.

Derek: Yeah. Right. [LAUGHTER] I’m in the southwest corner of the room and my partner is in the northeast corner. Sure. That could work.

Rebecca: We’re far away.

Derek: Right. I want to circle back to this question of why aren’t we just teaching online to begin with? And I think that’s a legitimate question. And I think it’s something that faculty and administrators have to really struggle with. What’s the value pedagogically of classrooms like this? Because a lot of it’s going to be really hard and awkward and, shall we say, sub-optimal. These are not the ways that we want to foster active learning and, like I said, the folks who aren’t fostering active learning, the folks who were just kind of, as the literature says, practicing continuous exposition by the teacher…. That, actually, is going to work no matter what you do this fall. If you’re fully online, if you’re hybrid, if you’re just going to do that, I would argue that that kind of instruction can work just fine online. And maybe that’s not a reason to have people in the room. So, why would you have people in the room? What is the kind of value added there? And one piece, I think, is that we’re all in this together, that this is challenging. I think we’re going to find some students this fall, who don’t want to be in the classroom, taking a health risk of some sort, encountering their instructors or fellow peers, and they’re going to really embrace the online option. I think you’re going to have some students who don’t want to have to stay where they are this fall, they need to come back to campus where they have reliable internet access and laptops that they can access and a library that they can access. We have a lot of students who, when they’re home, are not in conditions that are really conducive to teaching and learning. And so for some students, they’re going to actually welcome the chance to be back on campus and to be a part of that learning community again. And this is hard to talk about, because I think a lot of faculty have very strong reactions and opinions about what the fall is, and being required to teach online or being required to teach in person. But, I just want to put it out there, that I think our students are going to come at this from different perspectives. And so for some students that chance to come into class and awkwardly communicate a little bit with some peers may actually help them feel like they’re more engaged and more part of the learning community. I would also argue that, if we look at not just the individual class session, which may have this weird hybrid, physically distanced quality to it, but if you look at the semester, this fall, a lot of universities are announcing different calendars for the fall, they’re starting later, or they’re starting earlier. They’re finishing by Thanksgiving, they’re not doing fall breaks. We don’t really know what’s going to come this fall. And there’s pretty good odds that at some point, some campuses may need to pivot back fully online. And so if you think about designing a course for this fall, where you’ll have some virtual students, you may have students in vastly different time zones, you may have to pack the whole thing up and move it online at some point during the fall semester. Maybe you’re not assigned to teach the course online, but it’s still, I think, helpful to think about it as an online course, if you could design the course to really function well as a fully online course, and then treat your face-to-face component as a kind of add on, as a supplement, almost like a recitation section. So, there’s a little bit of this in a big lecture hall where you don’t have a lot of kind of student-faculty interaction, anyway. The recitation section is the kind of smaller space where you get to actually engage with peers more and talk about the stuff. And so if you’ve got a course that functions well, so that your assignments are online, a lot of your course communication is online, your key learning activities are online, but the face-to-face components, in this weird hybrid modality, are useful to that and supplemental to that, but if you had to give them up the course wouldn’t fall apart. I think that’s the way to think about this fall: as kind of online first, and then using the face to face to enhance what you can of the learning experience. Build the learning community, have that social interaction, give students a chance to practice and reflect on what they’re learning That’s still super useful. The other way to think about it, I would say, is maybe you’re not willing to kind of think about your entire course moving online right now. But, are there some key elements of your course that you can go ahead and move online at the start of the semester, so that if you have to move the rest of it online later, you’re in a better position. So, I think it was on your podcast that Jessamyn Neuhaus talked about having to learn how to do online assignment submission this spring; that she’d actually never gotten around to learning how to do that, which is fine. It was fine to have paper assignment submission up until the spring but then it became a requirement to do online. So this fall, make sure that you know how to use the assignment submission tool in your course management system and go ahead and plan on having students submit assignments that way. Make sure that you’ve got a good communication pathway with your students using email or the course management system or another tool like Slack or GroupMe, something where you can connect with students, maybe more informally. Go ahead and start using those tools from the beginning, so that if you do have to pivot fully online this fall, you’ve already got some essential components there.

John: That’s basically the approach we taken with our workshops here, and we’ve tried to help sell that to faculty, because it’s a bit of a lift for people who’ve only taught face to face before, by saying anything you create now is going to be something that you can use as a basis for future semesters of your course; that if you have these elements there, you can do a more flipped environment in your classroom, you can use your classroom for more active learning activities, and to the extent to which it results in more possibilities for active learning in the future, I think that’s going to be helpful.

Derek: Yeah, and I’ll add, we’ve been running an Online Course Design Institute at my teaching center all summer… every two weeks, all summer. We’re up over 300 participants in it at this point. A lot of faculty this spring figured out that online wasn’t necessarily as terrible as they thought it might be. They were able to connect with their students in meaningful ways and continue teaching in spite of the circumstance. And so we had a lot of faculty who woke up to some of the possibilities of online teaching this spring and then we’ve been working with faculty. he’ll spend two weeks with us in a pretty intensive institute, learning how to teach online, and a lot of them have a big shift in their opinion about online instruction over the course of those two weeks. They were initially skeptical that it could work nearly as well as face-to-face instruction. and they end the institute thinking, “Okay, this could be pretty exciting, actually, I see a lot of potential here.” And so that’s the other thing that I would suggest, that faculty keep an open mind about really the potential of online teaching. A really well designed online course can work just as well, sometimes even better, than a really well designed face-to-face course. And so it’s okay to kind of lean into that. And to let that be a bigger part of your kind of teaching toolbox this fall.

Rebecca: I think one of the things that you mentioned a little bit earlier in our conversation is like, why are you in person in the first place? What motivates being in person, I think you’re right about the social connection. even seeing other people who are also dedicating time to learning a particular thing could be useful, even if they’re not interacting with each other, and just in the same space at the same time. But also just if you’re there for equipment or other reasons, there might be ways of teaching using a lot of online techniques with the opportunity to have access to tools that they might not have otherwise. And it might be down to like access to a laptop or higher end technology or something that is in a lab or… I teach in a design studio, so some of the more expensive software, faster computers or things like that. So, we’re thinking through the ways that, maybe we don’t really need to be teaching so much, like there could be a lot of learning happening in the classroom at that time and not necessarily a lot of teaching… maybe some coaching and some interactions. But those interactions might actually be happening virtually,

Derek: Right. And you can imagine more of a kind of flipped model where some of the heavy lifting in terms of the teaching, the first exposure to the content is going to happen fully online through pre-recorded lectures or videos of some sort or other resources. And then that class time, as awkward as it is, is still an important part of having students apply things, practice things, get some feedback from someone else. That’s going to be a good model for the fall, I think.

John: Going back to something else you said earlier, the issue of the students who can’t be physically present during class time, there’s also the related issue of students who may not be able to be present virtually during class time if they become ill, or are remote and have limited computer access or bandwidth, or are in a different time zone. Would you recommend that faculty also start thinking about what types of asynchronous activities they can use to provide equivalent learning experiences for those students?

Derek: Yeah, that’s a good question. And again, I think this is the other thing that we’ve seen our faculty, most of our faculty don’t teach online. Our school of nursing has a really robust online program, but outside of that school, most of our faculty don’t have a ton of experience teaching online. So this has been kind of new territory for them. And one of the takeaways that many of them have from our Online Course Design Institute is realizing that you can do a lot of really valuable learning asynchronously online. For a lot of faculty this spring, online meant they had Zoom sessions with their students that essentially replicated what they would have done in the classroom face to face. But if you look at the last 20 years of online higher education, most of that work in higher ed has been asynchronous online learning. You build your course to work well asynchronously. And it’s only in the last couple of years that we’ve had the video conferencing technology that would make a synchronous online component something that you could really lean into in an online course And so thinking about some of your major learning activities and assignments and assessments, not just being online, but being asynchronous online, that’s a really good model to think about. I think one of the impulses is to say, “Well, I’m going to teach in this weird hybrid modality this fall, I’m going to have students in the room, I’m going to have students on Zoom at the same time, some of my students can’t make it during that time, so we’re going to record the Zoom session, and they can watch it later.” And that’s better than nothing, certainly, but watching someone else participate in class is not nearly as effective as participating in class yourself. And so I wouldn’t want faculty to just do that. That’s fine to do. But, I would want them to add something else intentionally to help those asynchronous students engage with the material. And it could be as simple as saying, “I want you to watch this Zoom session. And here are three questions I want you to answer by the end of that 50 minutes.” It could be a different set of questions than the students in the classroom are given to discuss… something a little more active to help them draw out some learning from those recordings. But again, it’s also fine to say, you know what, for this piece of learning, for this module, for this unit, the core learning is going to happen online asynchronously first. And for those who can attend the synchronous session, either in person or via Zoom we’ll do this supplemental piece. And so that’s okay, too. I just keep giving lots of options here. And I’m hoping that helpful. Faculty are gonna have to figure out what’s going to work for them and their students and their comfort zone. I also think faculty are going to have to learn to do new things this fall. During my town hall, I said, “This is going to be an exceptional semester. And so we are going to make some exceptional teaching choices. And that’s okay.” I think for most of us, 2020 has been suboptimal. There’s been massive disappointments in lots of ways and life has gotten harder in so many ways. And yet, we can either stay in bed and not try, or we can get out and try to make it a little bit better somehow. It’s this growth mindset. I think we need to approach the fall semester with a growth mindset to say: “This is gonna be hard. This is gonna be challenging. I’m gonna have to learn some new skills as a teacher. I may have to learn some new technologies. I’m going to try to do that in a way that doesn’t overwhelm me.” Don’t try to take on too much, too fast. But you’ll have to take on some new stuff this fall. And whether that’s active learning with technology in a hybrid classroom, or that’s designing an online course, or using some part of your course management system that you’ve never touched before, but might actually be helpful this fall, we’re all going to be stretching out of our comfort zones this fall. And that’s okay. It’s going to be hard work, but I think if we collaborate and lean on each other a little bit, we’ll be fine.

Rebecca: And it won’t be perfect. And that’ll be fine, too.

Derek: Right. It’s never perfect the first time out.

John: But with all these new tools, it can improve teaching effectiveness in the future. And that’s something we keep reminding people, that, yes, this is a challenge, but you’re learning a lot of new tools that have value beyond this. It’s not just for this one-time emergency, that this could result in some significant improvements in the effectiveness of your teaching later, even though it will be tough.

Derek: Yes, so one of the tools that we’ve been showing people this summer in our online course design institute is a social annotation tool like Hypothesis or Perusall. And it’s mostly our humanities faculty, but they love it. They are just over the moon with what they can do with these social annotation tools. And most of them just haven’t seen it before. It wasn’t on their radar. And it’s super useful in an online course. But, a lot of them are saying, “Oh, I’m just gonna make this a regular part of my courses going forward no matter how I’m teaching, because having students engaged with the text this way, where they’re annotating collaboratively and discussing it in the sidebar, that’s just a really useful learning process that I want to build into all of my courses no matter how I teach them.” So, we’re finding lots of things that we didn’t know were there that we’re going to make use of in 2021, and 21-22. These are going to be permanent parts of our teaching toolbox.

John: In our course redesign workshop for faculty, we included some samples and documents with Hypothesis, and people have been really impressed by the ability to engage and share and give feedback to each other. And I think we’ve got quite a few people who plan to be using it this fall. It’s a great tool. One of the things you recommend in this document is the use of a fishbowl technique. Could you talk a little bit about how that might work in this sort of hybrid environment.

Derek: Sure. And this is a technique that’s been in the literature for a long time, a way to foster discussion in the classroom. And someone mentioned this as a possibility for the hybrid classroom. And I was like, “Oh, yes, actually, that’s a perfect match.” The fishbowl technique classically works like this. You have a small group of students who have a discussion about whatever the topic is. They’re in the fishbowl. The rest of the students are observing from the outside and they’re quiet during the discussion. They’re taking notes, they’re observing. And then after the discussion, you then ask something of the observers, ask them to summarize what they heard or reflect on what they heard. And it can be really helpful if everyone in the fishbowl is advocating for one point of view. And then the folks who are observing have to then kind of summarize that, even if they don’t agree with it. It can really foster intentional listening. There’s lots of things you could do with the fishbowl, but when I thought about the audio context of these hybrid classrooms this fall, having some of your virtual students be in the fishbowl is totally practical. They’re the ones that are going to be easiest to hear across the entire class. You can have five or six students on Zoom, be the fishbowl, have the conversation, the rest of your virtual students and all of your in-person students are then the observers. They’re listening. They’re taking notes. They’re summarizing. I think that’s gonna work really well, actually. And as I’ve shared that idea with a number of faculty here, they’ve been excited to say, “Oh, yeah, that actually fits this context quite well as a structure for discussion.” And especially on some campuses, the virtual students and the face-to-face students are going to flip flop from day to day, there’ll be some students who come to class on Mondays and they do virtual on Wednesdays and the rest of the students are vice versa. And so you could have most of your students have an opportunity to be in the fishbowl at one point or another with this technique. And that way, you get to have some of the richness of that student-to-student discussion. It wouldn’t involve everyone at the same time. But, if you’re really intentional about what you ask the observing students to do with the discussion, I think it can be really productive. Because frankly, if you’ve got 40 students in the classroom, it’s hard to hear from all of them, anyway. You’re only going to hear from five to eight students in a typical discussion. This just centers them in a way and then guides the other students to participate well, in that type of small-group discussion.

Rebecca: I think what you’re pointing out here is the different ways people can participate in speaking isn’t the only way to participate.

Derek: Yeah, or like collaborative notetaking. This is something that a lot of faculty do as a matter of course, anyway, is have students have some shared document where some of the students in the classroom are taking notes on the class discussion. So their role in the discussion is different. They are not there to participate verbally, they are there to do the note taking piece. And that’s an important role. And that would work just fine in this hybrid classroom as well. And so part of this is thinking intentionally about how you want different groups of students to participate in the learning activities, and it’s okay to give them different roles and guide them to different ways to be meaningful participants.

John: You did mention collaborative note taking, wondering how that might be structured in a class of three or four or 500 students. Would it be reasonable perhaps to do that within your LMS using a groups tool to create that, having a shared google doc or something where you share it with a copy link?

Derek: I think if you’ve got 400 students in your class, that’s just a very different teaching context, and it is something about moving online. So I would say that having 75 students in a classroom and 300 students in the classroom, pedagogically, you’re going to use very similar techniques. If you want to foster active learning, then you’re going to have a lot of think-pair-share, a lot of peer instruction, you’ll have some polling. Anything over 50 is going to kind of look the same, pedagogically at least. Some of the logistics change when you have hundreds of students in the room. But, the kind of pedagogical moves that you’re making, I think are somewhat similar. Once you move online, I think there’s a much bigger difference between 50 students online and 300 students online. And so there’s almost a bit of the kind of MOOC mania that may be useful here. Right. So when we had massively open online courses that had thousands of students, there’s less difference between 300 students and 1000 students. And so we might even look to the MOOCs to see what are some techniques that work well at that kind of scale. And that’s where I think having an asynchronously design course makes a lot of sense. If you’ve got 400 students in the course getting them all together on Zoom is going to be a technical nightmare anyway. Let’s just make this course work well as an asynchronous course. The other piece that I would say that if you’ve got a big class and this gets to your point about collaborative note taking or other group structures that you might use, is that social presence is going to be really challenging. When you’ve got that many students, it’s going to be hard for students to feel connected. In the physical classroom, if you’ve got 400 students, at the very least, a given student has the five or six students they sit near every day to form a bit of a local learning community. And even if you don’t ask them to talk to their neighbor about something, they’re still going to talk to each other after class. And so there’s a bit of social presence, social identity, that comes just by virtue of the seating arrangements. In the online class, you’re not going to have that to fall back on. And I would advise instructors that have big classes to really think intentionally about permanent small groups, and to build in some learning activities and maybe even some assessment activities that leverage those permanent small groups. If you put students in groups of five or six students each and they’re going to meet with that same group every week, doing something useful during the semester, they’re gonna feel connected to the course a lot more than if they didn’t have that small group to lean on. We’ve seen this even in our Online Course Design Institute where we have 70 or 75 faculty participate over a two-week period, but we put them all in cohorts of size five or six. And you really get to know your cohort members, and what their courses are, what they’re teaching. And so that would be my recommendation for the bigger classes. And it could be collaborative notetaking. It could be every time you do a small group activity in class, you send them to the same groups so that they begin to develop working relationships with those group members, those are going to be really important for online classes that are large.

John: And you can always create Zoom rooms that have the same groups that you have within your LMS. So that way, the same students would be working together in both environments, synchronously and asynchronously.

Derek: Yeah, absolutely. You will have to learn how to do that. And again, we’re all going to be learning new tools this fall. I totally know that’s a thing you can do. I don’t know how to do that myself yet, but I would have to figure it out.

John: I haven’t done it yet, either. But I am preparing for one of those large classes in the fall. There’s a lot of questions I still have. One of the things I’ve been wondering about is perhaps the use of peer evaluation. I had done some of that earlier, but we had another tool that was specific for that. I’m not quite sure how well that will work within the LMS. And it’s a little scary at this point. But it’s something I am going to explore.

Derek: Yeah.

Rebecca: That’s something when I’ve taken a MOOC, even if I don’t feel overly connected to other people, there’s still some sort of peer evaluation piece where you feel like you’re getting peer-to-peer feedback, at least, even if it’s anonymous feedback, essentially, because you don’t know those individuals. And that can be effective in at least feeling like you’re learning with other people who are also learning. It’s not as effective as some other things, but it still does it a little.

Derek: Yeah, it does.

Rebecca: So we always wrap up by asking what’s next? [LAUGHTER] …and I don’t know, John seems to have me asking that question more frequently, because it feels really stressful to ask someone that right now. [LAUGHTER]

Derek: Well, I keep making the joke that it feels like March 97th. [LAUGHTER] Like, this has just been one long March. There was life before March, and now there’s life now… and time and space have no meaning anymore. So, next is a little ambiguous. What’s next? August…. August is next, right? [LAUGHTER] I can predict that. I think we’re going to have an August this year. [LAUGHTER] It’s really hard to kind of look beyond that. I would say, we’re focusing at my center on getting our faculty ready for the start of the fall semester. And what’s going to happen in the first three weeks of this fall semester, like, we don’t have that figured out yet. We don’t know what kind of programming we’re going to offer. We don’t know what kind of responsiveness we’re going to need. It’s been a lot this summer just to kind of do what we’re doing. And so it’s really hard to look very far out. I will say that, among many other complexities in the year 2020, it’s an election year in the United States, and more generally, we have a lot of protests that happened across the United States and across the world earlier this spring. There’s a lot of hard conversations that people are having right now, whether they’re pandemic related or not. And I don’t think that’s going to go away. I’ve been in triage mode all summer, trying to get faculty ready to teach online or hybrid. And so its been hard to think about all the things that may be challenging about this fall semester, but I do think the hard conversations that we need to have with our students and to help our students have productive hard conversations. That’s something that we’re going to spend at least a little time on in my teaching center in August, trying to help faculty get ready for what will likely be a contentious semester, regardless of the kind of modalities, the online, the hybrid, all that kind of stuff. Just the kinds of conversations that we want to have our students are going to be really challenging this fall. And so I think getting ready to do that well, it’s going to be an important component of what’s next for us.

Rebecca: And I don’t think any of us will be bored. There will be an August, and we will not be bored. [LAUGHTER]

Derek: Yes, those seem like certainties.

John: Well, thank you. It’s been wonderful talking to you and, we really appreciated the blog post as well as all the very many resources that you share on your website.

Rebecca: Yeah, thank you so much.

Derek: You’re quite welcome. We have a great team at the Vanderbilt Center for Teaching and I’m glad to be the Director and to get to share all the great work that my staff do all the time. So, thanks for that.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]