Category: authentic learning

Show Notes

• O’Neill Coffee Company
• IRACDA
• Martella, A. M., Martella, R. C., Yatcilla, J. K., Newson, A., Shannon, E. N., & Voorhis, C. (2023). How Rigorous is Active Learning Research in STEM Education? An Examination of Key Internal Validity Controls in Intervention Studies. Educational Psychology Review, 35(4), 107.
• Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the national academy of sciences, 111(23), 8410-8415.
• PlayPosit
• Panopto
• EdPuzzle
• Carl Wieman Science Education Initiative
• Clicker Resource Guide – from Carl Wieman Science Education Initiative
• Mazur, E. (1999). Peer instruction: A user’s manual. Pearson
• Knight, J. K., & Wood, W. B. (2005). Teaching more by lecturing less. Cell biology education, 4(4), 298-310.
• Deslauriers, L., Schelew, E., & Wieman, C. (2011). Improved learning in a large-enrollment physics class. Science, 332(6031), 862-864.
• Smith, M. K., Wood, W. B., Adams, W. K., Wieman, C., Knight, J. K., Guild, N., & Su, T. T. (2009). Why peer discussion improves student performance on in-class concept questions. Science, 323(5910), 122-124.
• Yabuno, K., Luong, E., & Shaffer, J. F. (2019). Comparison of Traditional and Gamified Student Response Systems in an Undergraduate Human Anatomy Course. HAPS Educator, 23(1), 29-36.
• Fagen, A. P., Crouch, C. H., & Mazur, E. (2002). Peer instruction: Results from a range of classrooms. The physics teacher, 40(4), 206-209.
• Lorenzo, M., Crouch, C. H., & Mazur, E. (2006). Reducing the gender gap in the physics classroom. American Journal of Physics, 74(2), 118-122.
• Lopez, J., & Shaffer, J. F. (2021). To Pre-req or Co-req: An Assessment of Why Chemical Engineering Students Elect to Take a Course as a Prerequisite or as a Corequisite. Chemical Engineering Education, 55(2), 86-94.
• Shaffer, J. F., Schriner, S. E., Loudon, C., Dacanay, S. J., Alam, U., Dang, J. V., … & Sato, B. K. (2023). The impact of physiology prerequisites on future anatomy and physiology courses. HAPS Educator, 22(3), 199-207.
• Stains, M., Harshman, J., Barker, M. K., Chasteen, S. V., Cole, R., DeChenne-Peters, S. E., … & Young, A. M. (2018). Anatomy of STEM teaching in North American universities. Science, 359(6383), 1468-1470.
• COPUS Protocol
• Casper, A. M., Eddy, S. L., & Freeman, S. (2019). True grit: Passion and persistence make an innovative course design work. PLoS biology, 17(7), e3000359.
• Shaffer, J. F. (2020). Student performance on and perceptions of collaborative two-stage exams in a material and energy balances course. Chemical engineering education, 54(2), 52-58.
• Recombinant Education

Transcript

John: Multiple studies have found that increasing course structure reduces equity gaps and provides benefits to all students. In this episode, we examine several ways to increase structure in STEM classes.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca:Our guest today is Justin Shaffer. Justin is the Associate Dean for Undergraduate Studies and a Teaching Professor in Chemical and Biological Engineering and in Quantitative Biosciences and Engineering at the Colorado School of Mines. He has taught a variety of both small and large STEM classes in multiple modalities using evidence-based approaches and has won multiple teaching awards as a result of this work. Justin is also an active researcher with 16 peer-reviewed publications and serves as the editor for four STEM education journals. He is the author of a forthcoming book on high-structure course design coming in late 2024 or early 2025 from Macmillan. Welcome, Justin.

Justin: Thanks for having me. I’m really excited to talk to you today.

John: We met at the POD conference back in November, and we talked about doing this but we finally got around to actually scheduling it. So thank you for joining us.

Justin: Thank you, John. Yeah, I saw you sitting there at a table by yourself. I knew your face from LinkedIn and I built up the courage to introduce myself and I’m glad I did.

John: Today’s teas are:… Justin, are you drinking any tea?

Justin: I am drinking some tea. I got prepared. It’s from a small little coffee/tea shop in West Middlesex, Pennsylvania. I’m from the near that area in western Pennsylvania. We call it the Shenango Valley and it’s called O’Neill’s and they make lovely Western PA coffees and teas and I just have some nice black tea here. It is that time of day for me that I need some.

Rebecca: Today I have a jasmine green tea, John.

John: Very good, and today I have a Prince of Wales tea, and a peppermint tea because I didn’t have very much to drink today, and my throat was getting really dry in my econometrics class. So I wanted to be well hydrated. [LAUGHTER]

Rebecca: Lots of variety, too. So we’ve invited you here today, Justin, to talk about how you’ve implemented high-structure course design. And for listeners who might be new to the concept, can you talk a little bit about what you mean by high-structure course design?

Justin: Yeah, so just like I’ll tell you all like I tell my students, I didn’t come up with this term. I wish I did. But I’m building off the work of other giants in the field. So as far as I know, the term really came out in the mid-2000s, 5-, 6-, 7-ish range from the University of Washington in Seattle. There’s some pioneering folks, Scott Freeman and Mary Pat Wenderoth in biology and they use this term structure to think about, “Well, how do we scaffold student learning before class, during class, after class, and we’re continuously building as we go up through the three levels.” So pre-class involves some kind of content acquisition, that can be reading your textbook, reading journal articles, watching Khan Academy videos, other sources like that. And then some kind of formative assessment before class because we all know students need a little bit of a carrot there to get some of that work done sometimes, which is totally great, because you give them a lot of chances, sometimes even unlimited attempts… again, formative assessment before class, make sure everyone has the same solid foundation, then you come to class, high-structure course design, just like any good class nowadays should be really rooted in active learning. So we got a lot of activities going on: group work, problem solving, case studies, real-world connections, any kind of flavor you want to do there with your active learning as long as students are engaged. And the big thing here though, is that you’re aligned to what happens before a class. We’re building off what happens before a class, I’m not in class just repeating what happens ahead of time. Rather, I’m using the foundational material students acquire before class, we build that up with another level in class, and then we keep going. And then after class, there’s a bucket there, where students are going to be practicing those skill sets they’ve been developing through the before class and in-class buckets, applying their skills to new contexts, solving new problems, doing projects, essays, more authentic assignments, whatever you want it to be. And then there’s typically a frequent assessment piece with a high-structure course design too, whether that’s a multiple midterm model, weekly quizzes, every other weekly quizzes, things like that, in addition to some more authentic assessments, which is something I’ve tried to get more into this past year, by adding more realistic projects and course assignments in my high-structure courses.

John: Could you give us some examples of some of the authentic assessments that you use in your classes?

Justin: Yeah, thanks, John. So this is really new to me still, this past year. I know I’m behind the curve. I know this is a term has been around there for a while. I always like to think I’ve had some authenticity in my classes for a long time. I integrate news stories, I use case studies, I have this new thing I called pod cases where I take podcasts and combine them with case studies and we use those in the classroom. But I thought I could try to be a little bit more authentic, even still… getting that advice from my wife for my personal life. She always says that can be more authentic. So, maybe I can make my courses and my assessments more authentic too. One example I’m doing right now. I’m teaching introduction to biomedical engineering, and I’m having the students write the actual legit National Science Foundation Graduate Research Fellowship proposal. Those of you familiar with that, there’s a personal statement part and there’s a technical kind of research proposal part. So we’re just doing that. It’s a two-page proposal, and we’re working on this in a structured way throughout the entire semester. Last week, students had to turn in the research topic and research question. This week, they’re turning in their hypothesis, in a couple weeks they’re turning in the literature review and experimental plan. We’re gonna have two rounds of a formal peer review of the entire proposal. And then we’re going to conclude with a mock panel discussion, just like you might have in the real NSF or the real NIH where students are going to be reviewing each other’s proposals and basically pitching them to each other and seeing if they should get, quote, unquote, funded or not. I don’t have any funds for them. Maybe we’ll bring candy on the last couple of days. But that’s kind of the idea of having a real thing that students could actually use some day if they’re going to graduate school. But even if they’re not, being able to write technically and persuasively in two pages on a proposal is a really important skill that they should be working on, in my opinion.

Rebecca: Sounds like you’re going to need some chocolate coins for that day.

Justin: That’s a great idea. I will get chocolate coins. [LAUGHTER] I didn’t think of that. [LAUGHTER]

Rebecca: Can you talk a little bit about what motivated you to attempt and move in the direction of high-structured course design.

Justin: I kind of started out this way, I kind of started in the deep end, if you will. So my postdoctoral program was called spire, S-P-I-R-E, and it’s one of these bigger programs called IRACDA through the NIH. They’re postdocs where you still do research. So I was a molecular biologist, biochemist at the time. I have a weird background in chemical engineering, bioengineering, molecular biology, but at the time, I was studying muscle proteins in squid, super fun stuff. And then I also got training, though, on how to teach evidence-based college science and engineering courses. In my postdoc, through great people like Ed Neil, Brian Ray Bozic, Leslie Lauria, and then also Kelly Hogan was someone that took me under her wing at the time, she was able to, along with the rest of the SPIRE team, helped me be introduced to using evidence-based course design practices, which included with high structure. So this was around 2010-11, right when these papers are starting to come out from Scott Freeman and Mary Pat Wenderoth. I started with high structure because the evidence shows it works. And that’s as a trained scientist, as a trained engineer, we like to go with evidence ideally to guide our decisions. And it’s no different with course design and teaching as it would be with thermodynamics or reactor design or anything else. The literature on high structure is really strong, even then it’s been growing over the years, some of my own data and a lot of other papers. The main gist, though, is that by having the scaffolding before class, in class, after class, students are able to really monitor their learning, develop self-regulated learning skills, be more metacognitive. And the data in a lot of papers shows all students are doing better in the class. But it’s not just everyone, using high structure and moving up in structure will start to close or reduce achievement gaps or performance gaps between historically underperforming students and their majority counterparts. So for example, female students and male students in physics, there has traditionally been a gap there, add some more structure to the class, those gaps tend to get smaller. And I’ve been participating in that kind of research here over the last many years with undergraduate students and my research team. And we found some similar results across different STEM disciplines. It’s really the evidence guiding me in these decisions to do it. I also think it’s a lot more fun. Once I have in mind one of my very first faculty interviews, oh a long time ago, now, I had an interviewer say to me, “Well, hey, I’ve looked at all the papers on active learning, and they’re all wrong. [LAUGHTER] All the statistics are wrong. What do you have to say about that?” And I paused for a second I said, “Well, even if that’s true, it’s a heck of a lot more fun to teach actively with active learning than not, I think the high-structure design model is more of a fun way to teach. And again, you’re helping the students with those study skills and metacognitive skills that they can then take forward to their future classes.”

John: There was a recent paper that did argue that many of the studies on active learning were flawed in some way in terms of a very strict methodological approach. However, there are so many of them, and the results are so remarkably consistent. I don’t really buy that argument in terms of the questions about methodology.

Justin: You’re right, it was a big deal, I think it was either Beckie Supiano or Beth McMurtrie in the Chronicle had articles on this going around for sure. And that’s the thing though, with us, as DBER folks… I want to introduce a new term, DBER – discipline-based education researchers… which I consider myself to be in. So I’m in the discipline. I’m in chemical engineering, I’m in biology, I’m in Biomedical Engineering, and I do the education research from that lens, but also from that training background. I’ve never been formally trained in education research methods in psychology or social sciences or psychometrics. I am, totally honest, I fake it. I do what I can. I know my boundaries. I know when I need a collaborator in the true social science disciplines that helped me with my studies. And so I think that paper was getting some of that. Sometimes the folks like us that are doing this work, whether we call ourselves DBER people, or Scholarship of Teaching and Learning (SOTL) people, we might not have that true background, and therefore the rigor of the paper, the methodologies, aren’t as advanced as maybe they should be. But like you said, there’s so many studies showing benefits of this, whether it’s simple correlational studies… sometimes they get to the mixed methods, even quasi-experimental studies, but the evidence just keeps mounting and mounting and mounting. And that key paper there was the 2014 one from Scott Freeman that PNAS meta-analysis showing that hundreds of papers, hundreds of studies… again, they have their flaws, don’t get me wrong… but you start to see these themes everywhere and you think, well, there must be an effect, something’s going on.

John: And we will include a link to that paper in the show notes, if anyone would like to follow up. But the fact that there may be these flaws in some of the research methodology suggests that maybe we need to see more people doing work in this area, and that there’s a lot of areas where there’s potential growth.

Justin: Totally agree.

John: And you’ve done quite a bit of research already.

Justin: I’ve tried, as a teaching faculty member now for this is going on year 12-ish, but I’ve always made that a part of my career. Before I was at Mines, I was at University of California, Irvine, where we did have a scholarship expectation or to earn tenure as teaching faculty. The UC system is one of the few in the country where you have actually tenure-stream teaching faculty, which was really cool. So I earned that, but I only had tenure about a month because then I left and came to Colorado, which… no regrets, I love it here at Mines… but I don’t have tenure here. But there’s still some expectations for scholarship, but not as high as it was in the California system. But I continue to make that part of my career, because, again, as a scientist and engineer, I think going based on data, based on evidence to guide your decisions, moving forward is a really important thing to improve what you have, because that’s all it comes down to really whether you call it SOTL or DBER or whatever, we’re trying to improve our courses, we’re trying to improve our students’ experiences. And yeah, so I’ve tried to do what I can over the years. And now I’m at the point where I want to help other people too. So I work with faculty individually, I do workshops via my side business, and even at Mines this semester, starting next week, I have a three-part getting started in DBER series kicking off for about 25 faculty on campus. I’m going to help them get started, get their feet wet in this field, and then we’ll see where it goes. Because I think a lot of folks, like you said, John, want to do this work, they’re interested in it, they might not have the time, they might not know how to, but you kind of give them the start, hold their hand a little bit and then see where it goes, and we’ll get even more evidence to support these claims.

Rebecca: I think a lot of faculty have an inclination to have a reflective practice and continuous improvement. They’re wanting to do SOTL work, but just don’t know how to formalize that practice. And we can certainly lean into helping people move in that direction.

Justin: 100% Rebecca, even when I started, I didn’t know what I was doing. If it wasn’t for Kelly Hogan and a few other folks at Carolina and then moving on from there at UC Irvine folks like Brian Sato, Diane O’Dowd, Adrianne Williams, they really helped me understand the field more. And then you start going to conferences and going to meetings and learning more about the tricks of the trades.

Rebecca: Can you talk a little bit about ways that a high-structure synchronous course might be different from a high-structure asynchronous course?

Justin: So I think the high structure while what I described earlier, you might be thinking of okay, a Monday, Wednesday, Friday class meeting at two o’clock in a lecture hall works great. I think it works even better in online for some cases, because you think about an online situation, whether again, it’s synchronous or asynchronous, your students you’re not really seeing and there’s that human connection of having eyeball to eyeball being able to understand, “Okay, I’m here I’m motivated, I’m ready to learn.” But whether it’s the recording, that’s the worst case scenario, when it’s totally up to you, or if it’s going to be on that live Zoom remote, where you can make your screen black, and who knows what else you’re doing. I’m guilty of that in meetings for sure. But having the structure though, having that scaffolding, having the clear deliverables of what is due when and seeing that step up from pre-class, in-class, after class, whether it’s the in-class is out again, synchronous Zoom session, or the recorded video you do on your own time, there’s that scaffolding helps students stay more connected with material, they understand more of what’s going on, when it’s due, what is due, how I’m being assessed, because that’s the nice thing about with structure. And just to be clear, I’m from the STEM disciplines, but you could do this any field. The research on high structure, as far as I know, is only been published in the STEM fields. I bet it’s being done in art history, I bet it’s being done in the languages, I just don’t know about it. So if anyone listening does this kind of teaching or research, I’d love to hear from you. But you can use that scaffolding again to help students be more transparent with the course. And that’s a word that I think comes through with high structure and is really true in that online environment, especially as transparency. I always tell my students, I want to know what you know, not what you think I want you to know. And by being uber clear with the learning objectives, starting with backwards design, just a clear Canvas website for the online cases can be really helpful for that. One distinction, though, with the async online versus synchronous online, is that kind of active learning piece, because we’re not there together. So one tip I like to share with faculty is that okay, if you’re recording a video for the in class, mimic, because you’re not actually meeting in class. What I always say is “okay, well make the video, but if there’s a point where you’d want them to answer a clicker question… I’m a big clicker guy… or if you want them to do an activity, just say, Okay, everyone, pause the video here. Take a couple minutes, think about this problem, what’s the answer or work on this, whatever it might be. When you’re ready, unpause the video and keep going. That’s a way that you can add the active learning to an online asynchronous class as part of the bigger high-structure model. I find when I have to record an online video, if I have traveling for a conference because I have to miss a day of class. I’ll have a 50 minute in-person class, when I make the video recording it’s only about 20 to 25 minutes, because all the active learning I don’t do in the video recording, but I give that tip… I tell my students, hey, pause the video here, answer this quick clicker question, solve this problem, come back when you’re ready. And so that’s a way to build the active learning into the asynchronous part, which again, is part of the overall high-structure piece.

John: And also, if you want to take it a little bit beyond that, you could use one of the tools where you can embed the questions directly in the video, and have them auto graded. I use PlayPosit for that, and it’s possible in Panopto, and EdPuzzle is another alternative that many people use. PlayPosit works really well. And I’ve been using it for a couple of years.

Justin: I just learned about PlayPosit maybe three weeks ago, I was doing a workshop at CU Boulder on clickers. And I kept hearing PlayPosit, it does seem like a very powerful tool to automate that active learning in the video piece, which I really like. I just haven’t had a chance to play with it.

Rebecca: So you did just mention that you’re a clicker guy, I believe. So do you use a flipped classroom approach in all of your synchronous classes? And how are you using clickers in your classes?

Justin: Yeah, I think that’s the technical term these days, clicker guy. So, I’ll go with that. I’ve been using them for as long as I’ve been teaching in various platforms. You name the platform, I’ve used it. I’ve primarily been using iClicker, just because that was the platform we had at UC Irvine and the platform we have here at Mines as well. Yes, I love using them. I think it’s really important. And you mentioned the flipped classroom design there, too, Rebecca. So when you think of high structure, you might think, “Oh, it sounds like a flipped classroom.” And it kind of is because we’re having students do stuff ahead of time. In class, they’re very active, but there’s still that structure piece even within class. I guess maybe I haven’t seen a truly flipped class because I think flipped classes, I think in class is like purely work time, purely activities, purely worksheets, homework type assignments, whereas I do a series of clicker questions, discussions before or after numerical problem solving, if it’s more of my engineering classes that I teach in thermodynamics, or material and energy balances, more conceptual sometimes if it’s biology or anatomy and physiology, but I’m breaking up a lot of activities and any talking with clickers. And so the clickers… I love all the modern bells and whistles that they have as that’s the another technical term I use, moving beyond multiple choice. And when I do workshops for faculty with clickers, that’s what I triy to get them to do. Okay, yes, you know, you can do A B C D E, but can you do multiple answers? Can you do target questions or hotspot where you touch the screen? Can you do ranking or ordering? numerical answers? …and I’m trying to get this traction on my own campus too. So even this semester, my first five questions of the semester, which I didn’t do intentionally, but after I wrote them, I realized, oh, I should do this every class now… my first five questions in my introduction to biomedical engineering class, each one was a different question type. So we did multiple choice, we did multiple answers, we did short answer, text entry, we did numerical, and we did target or touch the screen. And when I was done, I asked my students, “Hey, who’s ever used those other question types before?” And maybe four hands went up, I said, “Cool. What class did you do them in?” and they said, “Oh, your class last semester.” I seem to be the one and only that I know of on my campus that’s trying out these other tools, although like I said, I’m trying to change that. So yeah, if you’re using any kind of device like this, any type of ed tech, for that matter, make sure you justify it, justify the cost, and you make use of those bells and whistles in class. Or if it’s out of class, courseware, whatever it might be, just make good use of the tool, and you’ll find your students that are going to be able to buy into it and use it more effectively, and hopefully learn more from it.

John: And one option that we have here is a campus license, which actually is much less costly per student. And that reduces that barrier of students having to pay for it, which has led to a pretty rapid expansion in the use of it on campus.

Rebecca: One of the things that I really like about your example of the five questions is that extra level of scaffolding, of introducing all of the question types right away, and giving everybody an opportunity to practice those question types that you’re going to be using all semester. And so I just wanted to make explicit that other layer of scaffolding that you have built in there.

Justin: Yes, teaching students how to use the tools is very important. Thanks, Rebecca, for mentioning that, because whether it’s the app now on your phone for these clicker devices, or if it was the old physical device, there’s a learning curve with them. And same thing, even if it’s computational tool, or like Desmos, or something for a calc class. Yeah, you got to be able to teach students how to use the technology properly first, before you expect them to use it well. I will say though, with the learning curve of the clickers, when I was at Irvine, the app wasn’t really there yet. So everyone had the physical iClicker remote, and this was the iClicker w, so you could do text entry, and you could do number entry, but you had these up and down arrows, which you had to click to go through the alphabet. So I do the type of clicker question where I like to give four statements on the screen, and I say, “Okay, well tell me if they’re true or false, and you have to click in all four.” So you would click in a string of T’s and F, so it’s like TTFT, or FFFT, whatever it might be. And so when I would do this in intro bio there with 440 students, it sounds like a swarm of locusts, all the clicking to get to the T’s and then someone says, oh, you should change it. Just make A be true and B be false? I said “Great idea, I’ll do that.” But now with the app, we lose the sound effects and iyou can just type in T’s and F’s and it’s a little bit easier to use.

John: One thing I haven’t tried with iClicker, and I do plan to do it this semester is allowing students to rate their confidence in their answers. Have you tried that yet?

Justin: I did just last week. So I just found out that was available this semester, and I gave it a shot. And it was kind of fun. Like the first day I turned it on, they’re like, Oh, look at this, this question was 80% confident, and now it’s down to 40% confident and it kinda reflected the percent answers that were correct. So like the one that was 80% confident it was a single multiple choice answer. And it was pretty much most people got it right. The one that was lower confidence, like 40%, was a multiple select answer. So they had five options, but two or three could have been correct. And so the confidence goes way down through a different question types even. So, I think it was really fun for a day or two, and then kind of the novelty wore off, unfortunately. [LAUGHTER] So I’m going to try to revisit it maybe more in targeted times and targeted questions rather than doing it for every single one. But it was fun to try it. I’d encourage you to do it, too.

John: One of the reasons I was considering it is that it might help students improve their metacognition a little bit by reflecting on how well they thought they knew something and then getting fairly immediate feedback on whether their expectations about how well they knew it actually matched the response, [LAUGHTER] or the feedback to the response.

Justin: Absolutely.

Rebecca: When you’re using clickers, are you also using strategies like think-pair-share as part of your implementation?

Justin: Yeah, absolutely. And there’s that sweet spot of clicker responses that lead to that. So the best practices, I’d say, and a lot of this work came from Carl Wieman Science Education Institute, which I believe is still housed through Vancouver, British Columbia, but he’s at Stanford now. But there’s this great flyer they made like this trifold flyer that kind of walks you through how to use clickers properly, or best practices suggested, and then what are the possible outcomes? Typically, you’d start off with that silence. Okay, click in on your own at first, I tell them I want to hear crickets, just click in by yourself, and then I’m able to see on the app the distribution and so at that point, there’s typically kind of three outcomes. So one is maybe 80-90% have the correct answer. At that point I’ll say “Okay, let’s move on.” But I make sure to take time to have students respond and say, “Okay, well, why is it C?” or to have a little discussion about why it’s not the other answers. Because even though you get 90% correct, there’s still 10% of students who got it wrong. So we want to make sure that everyone’s on the same board. The next option is usually that kind of random distribution. Let’s say if it’s a multiple choice question, you get ABCD each are in the 20 to 30% range. At that point, I call that Hmmm, what’s going on? Was it a really hard question? Did I goof the question? Is there a chemistry exam this evening, so there’s some reason why maybe it didn’t click? And then what I call the Goldilocks case is when you get the kind of 50-50. So maybe you’re split between two answers. And I love actually, when it’s exactly 50-50, I’ll show that on the screen. But sometimes I don’t, and I’ll say “Okay, at this point, now, you’ve all had a chance to try on your own, now talk to each other.” So that’s kind of the think-pair-share that’s built into the clicker there. So this is peer instruction at its best. This was pioneered by Eric Mazur at Harvard, people like Jenny Knight at CU Boulder have studied this. And she actually has a Science paper with Carl and a few others that showed when students are doing this… So you give them some quick information on their own. Don’t show them the answer, have them talk to each other. They put a bunch of microphones around the lecture halls at CU Boulder, and they listened into their conversations. And they actually found students are legitimately teaching each other and debriefing the topic, discussing it, tearing it apart. It’s not just the one student says, “Oh, it’s A ,everyone else, click A.” They’re actually debating it and talking about it, so that you get that true peer teaching and peer instruction going on. And then typically, you see everyone moving more towards the positive correct response that you’re looking for, at which point you can say, “Hey, look, you don’t need me as your instructor, you’re teaching each other. You’re doing great. Kudos, keep up the good work.” Yeah, there’s so many ways to riff on clickers and use them in different respects. Whether it’s the more traditional question types, or now the more advanced question types, and no matter how you do it, you’re getting students more engaged, they’re having more fun with it. And fun actually is a word they use with clickers. I have a paper a few years ago from the HABS educator where I looked at clickers vs. Kahoot!s, and students rated Kahoot! and clickers both equally fun, where Kahoot! being the more gamified version with trophies and a podium and music and sound and stuff like that.

John: And both Eric Mazur and Carl Wieman and some co-authors had done some experiments on the use of clickers with that methodology with answering it individually and then discussing it and voting again. And they found some really substantial learning gains in both of those studies. And we’ll include links to those in the show notes. In all areas, we’ve been seeing some significant equity gaps, but I think since COVID, we’ve been seeing much worse equity gaps in terms of students because learning losses were pretty pronounced everywhere. And that’s especially in math and in the STEM fields, but they were especially large in lower-income communities where schools were not as well funded and where students did not have the equipment and so forth, needed to thrive as effectively in remote instruction. One of the things that some institutions have been using increasingly is co-requisites in place of prerequisites to help students who want to pursue a major in a STEM field, but who would otherwise fall way behind or take many years just to get up to the basic level of math needed for the entry level courses in the discipline. In 2021, though, you had done a study that did a comparison between the effects of prerequisites and co-requisites. Could you talk a little bit about that study and what you found?

Justin: Yeah, absolutely. We saw the same thing probably everyone else did with students struggling, returning from online instruction in high school and having online instruction here. I taught remote synchronous for three full semesters, even though at Mines we stayed open for business pretty much the whole time with in-person classes, just lower capacity. I had prior online instructional experience in my career, so I volunteered to do those online sections for our department. But with the co-rec/prereq issue, I think that’s really important to consider for a lot of reasons. And this work really started when I was at Irvine with Brian Sato and Pavan Kadandale and a few others, looking more in the biological sciences and “do prereqs matter?” It was kind of our big research question. And then I carried that over to Mines and applied that to chemical engineering too. The short answer is: it depends. So first of all, and kind of why do we even have prereqs or co-reqs? A lot of it seems to be more of a management thing. So if we want to control class sizes, moving from first year to second year, and so on, and so forth. But there’s definitely an idea of well conceptual understanding and carry forward for if you have Bio I, well, those skills you develop could be used and count for something when you get to Bio II or same thing with the calc series or whatever it might be. However, we don’t know. It’s unfortunate that I think programs don’t sometimes look at those connections, that vertical alignment, if you will, between courses in terms of prereq to the following class, or with the co-req issues. At Mines, what I did with the paper you’re referencing, John, in chemical engineering, we have these two courses. One is called material and energy balances, MEB, I’ll call it for short. And that’s kind of like our intro bio version for chemical engineering. It’s kind of like a survey course that covers a lot of different topics, gives you some foundational skills that you’re going to need as a chemical engineer. We also have introductory thermodynamics, and an intro thermo, it’s related to MEB, covers some same conceptual understanding and problem-solving skill development, but it’s more on the nature of energy, and that transformation understanding. So typically in chemical engineering curricula, and I’m getting in the weeds here, so apologies for anyone who’s tuning out cuz you’re not a chemie, but MEB comes first, that’s typically either the fall of sophomore or spring of sophomore, you take that. First, it’s kind of like your gateway class, and then that’s a prereq to intro thermo and everything else you take. At Mines though, we do a little bit different. Mines, we do intro thermo first, fall of sophomore year, and then in the spring of sophomore year, you take MEB and coming from that background myself as a student at Penn State, I wish I would have had that way because intro thermo is the more… I use the word gentle… it’s a gentler introduction to chemical engineering than material energy balances is. But the idea here is that you can take thermo as a prereq in the fall or as a co-req in the spring with MEB. So the natural question came up… it’s a nicely well organic experiment to test is, well, if you have the thermo as the prereq do you do better than if you take thermo as a co-req with MEB. And this has implications for advising and its implications for progression through our major through repeating classes, all sorts of things. So it’s a really worthy study and my student, Jordan Lopez, at the time, he was an undergraduate student in mechanical engineering, and I love working with undergraduates on these SOTL and DBER projects. Jordan came along with me for this we did a quantitative assessment to figure out okay, well if students in MEB, did you have thermo as a prereq or co-req? How’d you do in MEB? And then we also asked the students qualitatively “Okay, well, do you think having thermo as a prereq or co-req matters? And if so, what do you think? And what we found was there was actually a big deal. Students who took thermo as the prereq did significantly better in their overall course performance in MEB than if they took thermo as a co-req, and that’s using linear regression models controlling for incoming GPA and things like that. Students also felt the same way. They felt that if you have thermo as a prereq, you would do better than having thermo as the co-req with MEB, and again, the quantitative data bore that out. The reasons they gave for that was because they get some opportunity to get more familiar with the discipline, with how we approach chemical engineering, how we do problem-solving skills, and having that foundation in the prior class allows them to do better in the follow up MEB course. In this case, it mattered, prereq vs. co-req. However, in our studies at Irvine, with Brian and Pavan and others, we found it didn’t really matter so much whether you took for example, microbiology lecture before after microbiology lab, you would think those two would be highly correlated one as a prereq to the other, it didn’t really matter in the result. Same thing with anatomy and physiology sequences, it didn’t seem to matter too much, in other papers in which we cite in these studies looking at organic chemistry sequences, economic sequences, it’s kind of hit or miss with whether prereqs matter or not. Long-winded segment here, but I think we as programs should be taking careful look at our prereq sequences and core sequences in general to see what, if anything, our students carrying over from one course to the other, and maybe we should loosen that up a little bit, because if they don’t matter, that can help students be a little bit more flexible with their degree planning, and time to graduation and things like that.

Rebecca: Speaking of prerequisites, faculty are often complaining that students don’t necessarily recall [LAUGHTER] information from prerequisites as they go into a sequence like we might expect. In 2018, you conducted a study that investigated the effect of physiology prerequisites on the future anatomy and physiology courses. Can you talk a little bit about that study and what you found there?

Justin: Yeah, so that was what I was alluding to there, Rebecca, with some of the work I did with Brian and Pavan. And in those cases, there wasn’t too strong of an effect, if anything, with the sequencing and the ordering that students took these classes. At Irvine, we did have human physiology as a prereq to human anatomy. I know that sounds a little funny to those of you in the A&P space. But the reason we did that was because of kind of a controlling the size of things. Our physiology course there was a lecture-based course, it would have 3-4 hundred students in it every quarter… talking about over 1000 students a year taking this class, versus the anatomy class, which had a lab, we can only handle about 144 per quarter, so availability of seats in the human anatomy was much, much lower than physiology, and also not as many students wanted to take anatomy versus physiology. So that’s kind of why that ordering existed. And so we tested that, though, looking at with what Brian and Pavan and I came up with, we call it a familiarity scale. How do you do on questions that are based on how familiar you are with the content based on the prereq? What we did was we had the prereq instructors look at the follow up course exam questions and write them as very familiar or not familiar based on basically whether it was covered or not. And then we looked at student performance on those questions based on having the prereq or not, and we didn’t see much of a difference based on the very familiar versus not familiar questions. The way we designed those studies, we didn’t come up with any strong conclusions, again, on the biological sciences side saying that, yeah, these prereq sequences matter. But on the chemical engineering side, which I just described, that seemed to totally make a difference, the ordering of the classes.

Rebecca: It seems like it’s really important to think about why we’re structuring the curriculum in the way that we are. And is it just traffic control? Or is there actually a legitimate reason why we’re hoping that they are scaffolded in the discipline?

Justin: Yeah, I think it’s reality, a little bit of both that happens, which makes sense, kind of have that traffic control, but also making sure that that alignment occurs through our new programs or existing programs, and just taking the time as a program or department every so often to say, hey, let’s look back at this and see, why are we doing it this way? Things need to change in this modern day.

John: One thing I wonder a little bit, though, when instructors are complaining about how little students remember from the prerequisite courses that they’ve taken earlier, is how much of that may be due to the incentives provided to students. The way many courses are structured, not in the sense of the structure you were talking about, encourages students to do a lot of cramming before the high-stakes exams. And we know that that results in very little long-term recall. Might some of these issues be resolved if we could encourage students to adopt better learning strategies, and if we could design some of our courses at the lower level, perhaps, to build that in, using the type of structure that you’ve been talking about? Because at least from what I’ve seen, that’s not so common? You picked up this stuff fairly early, but I don’t think that’s part of the training of most of the faculty in STEM fields or in most disciplines.

Justin: Yeah, I agree with that. And there was even in 2018, I believe, in Science, a paper, I think they call it “The Anatomy of STEM Education in North America,” it was something along those lines, where they looked at a big survey a big swath of courses and trying to see who’s using Active Learning who’s not, they use the COPUS protocol, the observation tool to characterize what’s happening in the classroom, and they broke it down in these seven different buckets. And even at that time, a very small percentage of the courses surveyed in STEM were taught using these student-centered strategies, not necessarily high structure, but the active learning, student-centered ideas. But I completely agree with you, John, if we can get more faculty on board with using these types of evidence-based practices, which yes, it takes some time, takes some energy, takes some foresight to develop these courses. But you do the same thing with your research, it takes your time and energy and careful thought to put in your research experiments. We should put the same energy into our classes too. And if we can do that at the early stage, which again, there’s such a huge issue with loss of learning. Not only that, but also lots of students from the early stage. There’s a paper after paper for these bigger courses, taking them as quote unquote gateway classes, we get a lot of attrition at these levels especially of students who might not be as well represented in STEM. We can add high structure. We know that works. We know that helps students do better and succeed. It might not work the first time, don’t get frustrated everyone. There’s a great paper from Anne Casper and Scott Freeman about having True Grit with being able to try and try again. So in her case, she tried adopting high structure at a smaller school in Michigan, it took her about three tries to get the right formula for it to work for her students to see success. And that paper documents that trial and error over a few semesters. But if we can use these types of methods in our classes, ideally, our students are going to be better prepared then for future classes. I do these things called Reading guides, which again, another shout out to Kelly, I got a lot of inspiration from her at Carolina for these, these are just Word documents that I make. And I have a bunch of these on my professional website. If anyone’s interested to download them for free, go for it. They’re just Word documents that help students read the textbook and again be more transparent with what I want them to know. So it has them define terms, fill out tables, make little drawings, they answer checkpoint questions in the text, summarizing things in their own words. And with the reading guides, we’re able to help students again acquire that content and then use it in the class as needed. I’ve had students in follow-up classes say to their instructor, “Well, where the reading guides at?” And they’ll come to me like, “Hey, what are these things? What are they talking about?” So the point being with that story, though, is that I tell students that “Hey, even if you go to a follow-up class, where it’s not highly structured, you’ve developed so many great metacognitive skills, you’ve developed so many great self-regulated learning skills through this class that you can then apply it and add your own structure to follow-up classes, if needed. And these are the kinds of tips that I love working with faculty about to try to help them transform their classes when I visit a campus and do a workshop or work with faculty one on one or on Zoom or whatever, trying to take these practices, which again, are rooted in the evidence, they’re rooted in the literature, but we have to tweak them to your own personal situation, your own student body, your own courses and demographics and everything. You all know your students way better than I do. What I try to do is combine what I know from the literature and the evidence. And that says a lot of this is in the book I’m writing on high structure, it’s going to guide you through the course-design process, a lot of practical outcomes and deliverables. But you got to tweak it and match it with again, your unique circumstances to achieve those positive outcomes.

Rebecca: I think one of the nice things about introducing students to some strategies for high-structured learning and retaining information is that if you have things like your reading guide, and they show it to another instructor, now they might request that of their instructor and the other instructor might decide, “Hey, this is a good idea. This is a strategy I want to adopt too.” And that’s some of the best ways to get some of these techniques to spread.

Justin: It really is. It’s that camaraderie building through seeing each other’s practices, seeing what’s going on. I’ve been part of a project here at Mines since I got here almost six years ago now where we’ve been working on kind of revising how we evaluate, and what we consider effective instruction to be. So we’re trying to take a more holistic review of how we take into account the best practices of teaching and student learning. And one of the big pieces there is peer observation. So we’re actually rolling that out, as of this week and last week with doing departmental trainings and helping faculty see how we can do peer observation to learn from each other. I learned so much from going into another person’s classroom or looking at their Canvas site, or their syllabi or their materials. And I really got this from Malcolm Campbell, who was at Davidson College in North Carolina, when I was a postdoc, this is going back 14 years or so. And Malcolm told me once on the phone, he said, “Find someone who you consider to be a master teacher, and go watch him teach.” But he said “don’t go just once, go a lot. Once a week, if you can for several weeks, and you’ll get to see them handle different situations, you get to see the ups and downs of the semester, you get to see what it’s like on the day where the midterm gets passed back. Or you get to see what it’s like when they’re getting to a really difficult topic or there’s some kind of disruption that class and how they handle it.” That’s the one great thing about peer observation is that if I go to your classes, sure, I’ll be able to give you some comments or feedback on what I saw, but I’m going to learn probably more from seeing you two teach than whatever I could give you and then I’m able to incorporate that into some of my lessons and my teaching style. If we can just change that culture, make it a little bit more okay sometimes to talk about teaching, share teaching strategies, see each other teach and have fun with it really and learn from each other, I think that can go a long ways towards, again, improving student outcomes and helping them do better in all aspects of college and life.

Rebecca: I found one of the most fun opportunities for observations is doing observations outside of your own discipline. It’s really informative to see how people in wildly different disciplines approach teaching and the way that they operate their classroom. I remember visiting a dance instruction classroom when I was at Marymount, at my prior institution, and it was really interesting, it was fascinating to me about how specific and transparent every little move was and how to correct things and the anatomy that was being discussed and just the level of depth that you might not realize how something transpires in a classroom unless you’ve actually had the opportunity to observe it.

Justin: I’m going to play that some for my faculty Rebecca. I thank you, because that’s what we’re trying to do at Mines is not intra-departmental, but rather inter-departmental observation. So getting across the disciplines for sure. Because yeah, you learn so many different things. And even if you’re just going from mechanical to electrical engineering, there’s still a lot of differences between those types of courses, you might see what’s going on. And I’ve learned that myself, I’ve watched some economics instructors teach at Mines, no idea. I feel like I maybe should know this stuff to be a better financially literate person myself, but I learned so many cool ways about how they’re teaching, but I also learned some cool concepts.

John: We do have an open classroom project here on our campus, we keep hoping more people will open their classes and more people will attend. But the people who have participated have found it universally to be valuable. So we’re going to keep doing it, and we just hope to see it become more a part of the culture.

Rebecca: Sometimes just indicating to a faculty member that you’re interested in their content, and you just want to sit in because you’re curious is a great way to learn new content, and maybe get an opportunity to observe someone teaching. [LAUGHTER]

Justin: And yeah, I definitely pitch it that way. Like I want to learn from you. And you get some new ideas. And I feel like in that case, faculty are more than willing to have you in. Well, yeah, Best of luck with that open classroom project. I’ve seen those types of programs in the country kind of sometimes hit sometimes miss. But best of luck with that.

John: One of the things you’ve done is just some research or some experiments involving two-stage exams. Could you talk a little bit about how you implement them, and what you found in general? And how did students react to the two-stage exams?

Justin: Yes, so I’ll try to be brief on these because I could spend probably 45 minutes on these myself. But yeah, two-stage collaborative exams, or more affectionately called group quizzes or group exams for short. There’s been a ton of literature on this Jim Cook, UC San Diego, he really helped me get started in this about five years or so now. But it’s all disciplines. You look it up, and the one paper I have, you mentioned in the intro, there’s a lot of references to different STEM disciplines that use them and some non-STEM too. The idea, though, is that we push active learning, we push group work, we push collaboration, but then we assess you by yourself with a pencil and paper and a calculator on a piece of paper, so why not change the dynamic there a little bit. And the two-stage idea is that first you have students take an individual quiz or individual exam, and I personally have gone to a weekly quiz model with my high-structure courses. I don’t do midterms anymore, I don’t do finals sometimes even. I know that’s taboo sometimes, but it depends on the class and I might not even do that. My colleague in Irvine Adrienne Williams says with high structure by going to the weekly quiz model, instead of having these giant peaks of anxiety with the two or three midterms, we just have a smooth level of anxiety the entire semester, and spread it out a little bit. But, for me, group quizzes then, what they look like is students take the individual quiz, turn it in, then they get in a group, this is a predetermined group, they know who their group is going to be. And then I give them a group quiz. That group quiz in the paper that you referenced was an identical one. So they took the identical two problems. And this was a MEB material energy balance class. So it was just two numerical problems, problem solving. They did that again as a group. However, my other courses now I do a different quiz. It’s either an isomorphic question, so slightly similar, but a little bit different, maybe in intro thermo or if it’s biology or anatomy and physiology, completely different set of questions. Then they do it as a group. The reason we do this, again, was trying to get this collaborative piece in the group setting. But also, when do you find students are most excited to talk about a quiz or an exam? It’s as soon as it’s over, they go walk out, the hallways abuzz, everyone’s all excited, good or bad excited, but they’re talking. So let’s capture that energy and do it in the classroom through that group assignment. Now, the group quiz is usually a fraction of the weight. In the literature, you’ll see, it’s usually let’s say you call the whole experience 100%, the individual quiz, maybe 80%, the group quiz is 20%. But that can vary from 75 up to 95, however you want to do it. And again, they’re covering the same kind of concepts just in that group dynamic. And so when I did this the first time, I got some consultations from my co-instructors, Rachel Morrish, Dave Marr at Mines, and I was a little bit nervous to see how it would go. But it was a hit. It was an absolute blast. The energy in the room is one thing I cannot describe it strongly enough here through audio, it’s just you got to see it. The students are energized, they’re talking, they’re debating. They’re working through the problems together. They’re trying to figure out, I will use the F word here. It is fun. It is a very fun experience to see. I’ve had students tell me that after the quiz is over, like that was a lot of fun. Talking about spreading things on campus. This is one thing that has spread a little bit at least within my department of chemical engineering, a few other classes have been using them now. And I collected some data on this, which I published, showing that… which makes sense, there was no shock… but the group quiz score is higher than the individual score because you’re tackling this together right away. And then also though, students strongly preferred it. The affect, the positive attitude response was through the roof. Both of those results, these are replicated from other disciplines, they see the same kind of things. I think it’s a really nice way to promote that community in the class, show that you’re working together, add that active part to the assessment. However, I’ll end with saying there can be some resistance to this, especially from students who might feel like “hey, I can do it all on my own, I’m fine.” And then they have to work together with maybe some students, they don’t know as well. You might get some resistance from students saying that I don’t want to work with someone else, I wanted this on my own. So be prepared for that. Also, accommodations with testing, if you have students who get extra time, that can be difficult to manage. I will say though, from doing this now about five years and four different classes at Mines, I’ve never had a single issue with students and extra time. What I typically do is I’ll have students get the extra time for the individual part, then they come join us for the group part and have the normal amount of time because in that group setting the students with extra time, they’ve always told me and I always ask them, I say, “are you okay with this?” And they’re like, “Yeah, it’s fine to have the normal time because I’m in a group.” So just some things to be aware of, if you’re going to try them out. And again, I could go on and on with these, so if anyone wants to learn more, hit me up. I’m happy to chat with ya.

Rebecca: Sounds like a good day to visit a classroom, huh?

Justin: It’s an energetic day for sure. Although you might not see much teaching going on if it’s just the assessment part. But it’s fun to drop in and see what’s happening.

John: There is a lot of teaching going on in their second stage, though.

Justin: Absolutely.

John: In fact, when I first did it, I was so excited about it, I made a short video recording and sent it to Rebecca, so she could see the dynamics of that interaction.

Rebecca: And hear it.

John: It was just really, really different than going over a quiz, where some of them would be really happy, but not that focused. And the students who did poorly were generally pretty disengaged when we went over a quiz ift they didn’t do so well. But when they were explaining it to each other, it was so much more positive.

Justin: Yeah. And you’re right, the teaching is happening. It’s coming from the students to each other. Not for me necessarily, but it’s coming from them. And it’s so powerful to see… absolutely to hear that live.

Rebecca: But you’ve set up the structure to facilitate it. So therefore, you’ve done the teaching.

Justin: [LAUGHTER] Thank you. Yeah.

Rebecca: So we always wrap up by asking: “what’s next?”

Justin: Yeah. So for me, what’s next? You mentioned at the top, John, I have this new position, I’m Associate Dean of Undergraduate Studies. And I mentioned in passing, I have a side business I called Recombinant Education. I do workshops for faculty. I’m writing a book. For me, I don’t know what happened, but last year, I had a birthday the ends in zero. Things have been a little wonky in my brain lately. And I listened to Taylor Swift now, sometimes. I never used to, but I think a lot of people do that. So it’s not just me turning that age. But the point being is that I’ve been doing this enough time now that I feel like I have things to share. And I’m trying to do more of that. So I’m trying to, and I’m still nervous to do things like this kind of stick my neck out and even talk to folks like yourselves, but I want to share best practices with teaching and learning, specifically with STEM education, because I still hear from students and I still see it from time to time, students are just not getting the best classroom experience they should be getting. College keeps getting more and more expensive. Students don’t deserve to have less than excellent experiences, in my opinion. I’m doing what I can on my own to share these best practices. Someone called me a one man teaching and learning center, it’s probably a fair way to say it. One of my students said, “Oh, you’re teaching teachers how to teach,” and that’s, I guess, another fair way to say it. But again, I got the energy for it now. I kinda have this midlife crisis going on too where I want to make some bigger impacts on the world outside of my own bubble, whether it’s at the administrative level at Mines now or across the country, or even the world. So I love working with other faculty on all these different areas. And that’s what I’m trying to make a big part of my day-to-day life now. But I’m still in the trenches. I’m still teaching classes every semester, because that really keeps me rooted, lets me try some new things out, still doing the DBER work and the SOTL work, working with students and just trying to have a blast and do what I can here in the time I have left.

John: Well, thank you for joining us. And we’re really looking forward to that book. And I hope you’ll come back and talk to us again when the book is close to coming out. Or sooner.

Justin: I would be honored to come back anytime John and Rebecca and if both of you are ever in the Denver golden area, please let me know. I’d be happy to show you around out here.

Rebecca: Well, thank you so much for joining us. It was fun to talk to you and to hear about the fun happening in your class.

John: More F words.

Rebecca: Yeah, [LAUGHTER] a lot of F words.

Justin: Yes. [LAUGHTER] Thank you both. I appreciate it. It was fun.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

GANESH: Editing assistance by Ganesh.

[MUSIC]