Students with strong critical thinking skills can make better decisions that have real consequences on their quality of life. In this episode, Michael Bishop joins us to discuss an approach to developing these skills that appears to have a persistent effect on students’ ability to avoid common cognitive biases.
Michael is a professor of philosophy at Florida State University. His research interests focus on two areas: rationality (how should we think about important things) and happiness (what is lasting happiness, and how can we get it). He is also a co-author of a recent study investigating the impact of critical thinking classes on common cognitive biases.
Show Notes
- Bishop, M., Feltz, A., & Conway, P. (2026). Critical thinking classes can reduce common biases: Results from a field experiment. Journal of Experimental Psychology: Applied.
- Open science framework website for this study
Transcript
John: Students with strong critical thinking skills can make better decisions that have real consequences on their quality of life. In this episode, we explore an approach to developing these skills that appears to have a persistent effect on students’ ability to avoid common cognitive biases.
[MUSIC]
John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.
Rebecca: This podcast series is hosted by John Kane, an economist…
John: …and Rebecca Mushtare, a graphic designer…
Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.
[MUSIC]
Rebecca: Our guest today is Michael Bishop. He is professor of philosophy at Florida State University. His research interests focus on two areas: rationality (how should we think about important things) and happiness (what is lasting happiness, and how can we get it). He is also a co-author of a recent study investigating the impact of critical thinking classes on common cognitive biases. Welcome, Michael.
Michael: Thank you. Glad to be here.
John: Thank you for joining us. Today’s teas are:… Michael, do you have tea by any chance?
Michael: No, not today. I usually drink tea in my office, and when I do, it’s usually a Chinese black tea.
Rebecca: Always a nice choice. Today I have an English breakfast tea, nothing that exciting,
John: And I just have nothing that exciting either, a Lady Grey today.
Rebecca: We got to step up our game, John. So, Michael, we’ve invited here today to discuss the field experiment that is described in your co-authored paper entitled “Critical Thinking Classes Can Reduce Common Biases: Results from a Field Experiment.” But before we discuss the study, can we start a little bit with the concept of critical thinking? You teach critical thinking. What are you trying to teach? And what makes someone a good critical thinker?
Michael: Yeah, I think that’s a great question. One of my favorite sayings is the philosopher George Santayana said that a fanatic is someone who, having lost sight of their goals, redoubles their efforts. And so, in teaching critical thinking, I think it’s important to start with an idea of what you want to teach. And so there’s a lot of possible answers. Some people think that a good critical thinker is someone who uses certain rational methods or rules like logic or probability for the scientific method. Other people think that a rational thinker would be an expert debater, someone who can detect fallacies and build a coherent case for their views. Other people view good critical thinkers as virtuous thinkers; someone who is careful, open-minded, curious, intellectually honest, intellectually humble. And I actually think these views are either wrong… so I think that good debaters are often poor reasoners… or incomplete. They capture part of good reasoning, but not everything. So basically, what I am trying to teach in my classes is teaching people to ask good questions. Ask what I call fruitful questions, questions that help us get better answers, not killer questions that end debates, but questions that, if you’re trying to answer them, help get us better answers. So when you view that as the goal, how you go about teaching critical thinking becomes much easier.
John: And as you suggested, people have many different descriptions of what critical thinking is. I know whenever we talked about the importance of this in our Gen Ed program, everyone agrees that critical thinking is extremely important, but they all disagree as to what exactly it is. So it’s helpful, I think, to start with that statement. In this experiment, you’re focused on five common biases. Can you describe the biases that were selected for this study and provide some brief examples of those biases for our listeners?
Michael: Sure. The way psychologists think about these is in terms of biases. The way I think about them is in terms of cheat codes. So there are certain cheat codes people use in debating or in arguments, and cheat code arguments are arguments that you can use to defend pretty much any view you want. And so cheat codes are really tools of manipulation, more than tools of trying to figure out what the truth is or what a good decision is. And so when you think about good critical thinkers as thinkers who can ask good questions, the way I think about it is when you detect a cheat code, when you detect somebody using a cheat code on you, you should be able to ask them a question that diffuses the cheat code and points them in better directions. So let me tell you some of these cheat codes. One I call the focalizing cheat code, and it might go something like this: We should go to war because it has certain benefits, so, if an argument like that, or we should not go to war because it has certain costs. If you’re convinced by an argument like that, I think that you’ve fallen for a cheat code, and the best way to see this is to consider an example where where it’s clear that it’s a cheat code. So, suppose I gave you two envelopes, and I said you can look in these envelopes, and you can take all the money that’s in the envelope that you choose, but you can only choose one envelope. Suppose I open one envelope, see that there’s $5 and I take it without looking in the second envelope. In that case, you’d think that I’ve made a mistake, that I haven’t reasoned wisely about this particular decision because I didn’t look in the second envelope. If I say, “Look, the reason I chose the first envelope is that having $5 is better than not having $5.” It’s true that having $5 is better than not having $5, so there is an advantage to choosing envelope one. But when you think about it, what makes a choice a good one isn’t that it makes you better off than you are now. What makes a choice a good one is that it makes you better off than what else you might have done. Now, economists call this opportunity cost, as I suspect you guys know. And so, one thing I’m trying to do is get my students to, when they see someone argue for, “Hey, we should go to war because it’s going to bring us benefits.” I want them to at least stop and be able to ask the question, wait a minute, what are some alternatives to going to war? Because even if it’s true that going to war brings us benefits, and we can argue about that, but even if it’s true, it might not be the best thing to do, in the same way that choosing envelope one, even though it brings us benefits, might not be the smart thing to do. So I call that the focalizing cheat code. Another cheat code that I teach is what you might call the correlation cheat code. So I say, “Hey, a new healthcare plan is passed, and healthcare costs have gone up 7% a year since this plan is passed. Boy, this plan really stinks. It’s caused our healthcare costs to go up a lot.” This again, the question isn’t is the conclusion right in the war case or in this case, the question is: Is this a cheat code argument? And it is. And the reason is, what you don’t know, and what you really need to know, is how high would have costs been had this healthcare plan not been in place. So again, I say this healthcare plan is great or it’s bad, and I point to something that happened afterwards that. That’s what I call the correlation cheat code because you’re taking this correlation and you’re taking it as evidence for thinking that this policy has caused this particular result. And so again, what I want people to do is just stop and ask the question: “Well, wait a minute. Maybe without the health care plan, costs would have gone up even more, or maybe they would have gone up just the same, or maybe they’d have gone up less.” But unless you ask the question, it’s easy to get manipulated into accepting views that aren’t true. Another cheat code is the regression cheat code. This is a very common one. So lots of coaches and parents think that punishment is effective at molding behavior. I know that when I played sports as a youth, my coaches thought that after a poor performance, if you punish the players by making them work really hard in practice, they’ll do better next time. And sure enough, after a poor performance and after we were punished, we did tend to perform better. But again, the question is: “Did the punishment do that, or was it just a return to our normal level of performance?” So that’s the regression cheat code, rather than just saying, “Hey, maybe you’re returning to your normal level of performance,” saying that “it’s the result of a punishment or the result of a reward.” The third one is sunk cost cheat code, another one that’s a favorite of economists, defending a decision because of the resources already spent on it, so let’s stick with this policy or this war because of sunk costs. Those sunk costs are gone; they’re gone no matter what you do. So you might as well do what’s best for your future. And the last one that we looked at was the gambler’s fallacy, and that is basically treating statistically independent events, as if they were statistically dependent. So, if a coin comes up five times heads, and it’s really a fair coin, thinking that it’s more likely to come up tails next time. So, those were the five cheat codes we looked at.
Rebecca: I think the soundtrack that we have going on with the thunder is really emphasizing that these are cheat codes. [LAUGHTER]
Michael: Indeed, indeed. [LAUGHTER]
Rebecca: It’s very ominous. [LAUGHTER] Can you talk a little bit about how you chose these particular cheat codes?
Michael: Sure. So this goes back to my view about good reasoning. So 20 years ago or so, me and a friend defended a theory of what rational reasoning was, and what we said was that the good reasoning is reasoning that’s cheap, it’s accurate, and it’s focused on significant problems. And so the way I chose these cheat codes is because I thought that they were significant. That is, people use them to manipulate you when it comes to really important issues. So I see these cheat codes a lot out there being used in the world for really important issues. And the cheap part came in because I think that in thinking about how to teach critical thinking, I think one thing you want to do as a critical thinker is to diagnose the problem. Why are my students thinking badly when they do think badly? And I think there’s three general answers that people tend to think about, although often this doesn’t get talked about enough. So one reason students might be thinking badly is that they don’t know these rules, so it might be an ignorance problem. And if it’s an ignorance problem, then it’s going to be really hard to teach them to use these rules in a way that they keep using the rules after they leave the class. Because not only do they have to learn the rules and learn to apply the rules, but when they leave your class, they need to stop thinking in their old ways and use these rules, which tend to be hard to remember, hard to apply in the real world. And so, one reason that I think studies have not been successful or have had limited success in teaching people to reason well in a way that sticks once they leave the classroom, is that too often we teach critical thinking as if the problem is an ignorance problem. So we need to teach our students new rules. So one reason I chose these rules is that I speculated that they already knew these rules and they sometimes applied them correctly. And if that’s right, if they sometimes apply these rules correctly, then I don’t need to teach them a new rule, and I don’t need to teach them how to apply the rule. They’re already doing that. All I need to do is get them to see that it’s the right rule, and then ask them, “Hey, why aren’t you using it in these cases over here, where it seems like you should still be using the rules.” So, for example, think about the two envelope problem that I talked about before. None of my students think, “Oh, yeah, choose the $5, that’s the way to go. None of them think that. They know perfectly well that the way you evaluate that choice is not in terms of “Hey I’m $5 better off than I am now,” it’s “Hey if there’s $1,000 in the other envelope, then I’m 995 dollars worse off than if I’d have chosen envelope number two.” They all see that, so I don’t have to teach them any new rules. All I have to do is when I say “Hey let’s go to war, it has this advantage,” and if they say, “Yeah, that sounds good,” I can say, “Wait a minute. How is this different from the two envelope problem?” So the main reason I chose these, and these aren’t the only rules I teach in my critical thinking class, but the main reason I choose the rules that I do is that I think that my students already know these rules and at least sometimes apply them, and so when I introduce them to these rules in cases to which they are inclined to apply them, they give me what I call the old bovine stare. They look at me like, “Well, that’s obvious. You’re not teaching me anything I didn’t already know.” And that’s what I want. I want them when I introduce these rules to say, “I already knew that. Why are you teaching me something so obvious?” Because then the next class period, or a week from now, or a month from now, when I introduce an argument where they ought to be using this rule that they said was so obvious and they’re not, I don’t have to say, “Hey, you guys have made a mistake.” I can just ask them, “Okay, wait a minute, you said this rule was obvious, why aren’t you applying it here? Either you were wrong on day one about that rule being the right rule, or you’re wrong right now in not applying that rule.” And so there’s a real sense in which the reason I chose these rules is because I think I’m really not teaching my students anything new. So here’s one way to think about it: rather than teaching them like calculus, like a chain rule, like a new formula that they don’t know, what I’m really trying to do is get them to recognize cardinals. They already know what a cardinal looks like, so I’m just trying to get them to recognize cardinals, and so that’s why I think that the way I teach critical thinking tends to work and is way easier than starting off with the assumption that “Hey, my students don’t know anything, and I need to teach them a whole bunch of new rules.” So that’s why I chose these rules, and that’s why I choose the rules that I use in my critical thinking class.
John: And these are rules that, when you think about them as you framed it, they’re pretty obvious. But in practice, they’re rules that people violate all of the time in their decision making. What really impressed me about the study was that you did get some [LAUGHTER] results out of this because so many studies of behavior modification have been found to be ineffective beyond a very short period of time, often not lasting through the end of the class.
Michael: That’s absolutely right. In fact, one of my old professors, he said this about logic classes, but I think it unfortunately is often true of critical thinking classes, but he said that logic classes teach you to do one and only one thing, and that’s take logic tests. And I sometimes worry that that’s the case with a lot of our attempts to teach people to think better, is that what we’re really doing is teaching them to take tests because they will use the rules on the test. But if you want them to think about this afterwards, I really do think that the main thing is to get them to commit to the rule, and then get them to recognize, “Hey, wait a minute,” to be sensitive to when they see it in the real world because it’s all over the place. Once you get sensitive to it, you start seeing it everywhere. And indeed, one danger of my class, of my critical thinking class that I talk to my students is what I call “the turd in the punch bowl problem,” and that is being a good critical thinker isn’t necessarily going to make you popular, and so you need to be careful about when you point out to people that their arguments aren’t very good, because if you do it too much, you will end up being like the turd in the punch bowl at a party. People will look at you and say, “Oh, I’m going to stay away from that person.” And so, at the end of the semester, I will ask them, “Have you had any turd in the punch bowl experiences as a result of this class? And at the end of the semester, a lot of the students – and I don’t know if this is good or bad – but about half of the cases involve the sunk cost principle, and usually it involves the destruction of a relationship. So people will end a relationship because the only reason they’ve been sticking with the relationship is all the pain and effort and heartache they’ve put into the relationship.
John: And I know behavioral economists have noted that quite a bit. That once they really get this message through, it leads to a breakup of so many relationships of their students. But even people who know these things sometimes fall into these traps. And one example I think I mentioned on the podcast before, and I know I’ve mentioned it to Rebecca, is one time I was at a conference with a few other people, and we went over to Universal park, and it was just really hot that day, it was like 97 degrees, and we had spent quite a bit for the tickets for the day, and we were just kind of miserable. And one of the other people said we should just go. And my response to a historian was that we spent $230 and we’ve gone on two rides. We really should stay and get our money’s worth. And her response was, “But that’s a sunk cost. It shouldn’t matter,” and it’s easy for us to fall into that.
Michael: Absolutely, and indeed, this is one of the reasons I choose the rules that I do, is that I think many of them are significant. They really can make a difference to the quality of your decisions and the quality of your judgments.
Rebecca: Can you talk a little bit about the two-stage training program that was used in class?
Michael: Sure. So I gave you a little bit of an example of it already, but let me give you an example with another case with the correlation cheat code. So my go-to example here is I’ll tell the students, “Well, you know, yesterday I ate chocolate and today I broke out with zits. Boy, that means that eating chocolate causes zits,” and they immediately say, “No, that’s terrible.” So the first stage, what I’m doing is I’m introducing examples where they are disposed to use the rule. So nobody says, “Yeah, that’s great evidence for thinking that eating chocolate causes zits.” And then we talk a little bit about, “Okay, well, what would count as good evidence?” And they come up pretty quickly with maybe just a little bit of help, “Well, you want to know what happens when you eat chocolate and when you don’t eat chocolate, because what you really want to know is what difference does eating chocolate make to your complexion, and so what you want to know is what happens when you eat chocolate and you don’t eat chocolate, and everything else relevant to your complexion is the same, and what difference is there in these two cases,” and so that’s the first stage. The first stage is just introducing them to these rules, these cheat codes. So I give them the cheat code, but really a cheat code which they detect right away as a cheat code, and that they know as a cheat code. And then I spend the rest of the semester giving them arguments that use that cheat code, but are not as easy to detect. And often these are political arguments, and one reason I don’t get into trouble – at least I haven’t so far – in giving political arguments is that these cheat codes you can use them to defend views on the left, on the right, in the middle. They’re tools of manipulation. It doesn’t matter what the truth is. You can use these cheat codes to defend the true view, the false view, smart decisions, dumb decisions. And so, what I’m really interested in doing is getting them to be able to detect these cheat codes. So a first step is just introduce them to these cheat codes in a way that they will be disposed to accept and think is obvious. And then after that, I’ll give them lots of examples in which they’re, at least many of them, are not disposed to think is obvious. And then rather than me sort of coming from on high and saying, “Well, you know, you made a mistake there, dummy.” I just step back and I say, “Wait a minute. We seem to have an inconsistency here. You insisted that this rule was right on Monday, and on Friday you’re not using the rule in this case. What’s going on?” And I have yet to have a student, and I’ve taught 1000s of students, I’ve yet to have a student say, “Okay, I’m going to rethink that rule.” They all say, “Oh, well, I made a mistake here. I reasoned badly,” and by the end of the semester, they’re not reasoning badly.
Rebecca: Sounds like you spend a very small amount of time on stage one and significant time on stage two. Do you do all of the cheat codes at the beginning of the semester and then work on utilizing them throughout the semester, or do you kind of do units on different cheat codes? How do you organize?
Michael: So I will introduce a cheat code and then I’ll give them examples using that cheat code and all the cheat codes we’ve covered. So once a cheat code is introduced, they might see it at any point in the future, and I think it’s really important they keep practicing it and keep being able to recognize it, and not just think, “Hey, I know this is the cheat code we’re working on today, so I just need to be on the lookout for that one.” Another thing I do, and I think this is really important, is I will sometimes give them arguments that are promising, that if you grant the truth of the premises, they at least give you some reason to accept the conclusion. Because I don’t want the students just to be skeptical of all arguments. I want them to be able to distinguish between “Okay, that’s a cheat code. That’s definitely a bad argument. The conclusion might be true, the conclusion might be false, but that’s not a good reason to accept anything” and then “Okay, well, what would count as good evidence for thinking that this is a smart decision or that this policy has led to increased costs or decreased costs?” And I want them to be able to say, “Okay, look, that argument right there, that’s pretty promising. This argument over here definitely uses a cheat code.”
John: So you start off with a little bit of retrieval practice with some applications of the same concept, and then you move into some interleaved practice, which raises the cognitive difficulty just a little bit. But it’s one of those desirable difficulties where the more cognitive effort that students have to place in identifying what is at issue, the more deeply they tend to learn it, and the more likely it is that those concepts will stick and also transfer into other applications, which is what you’re building to in this program.
Michael: Absolutely, absolutely, the interleaved practice is really important, and I wouldn’t go back to sort of the blocking practice.
Rebecca: So we’ve talked a little bit about what the intervention is. Can you talk a little bit about how the study was conducted, like the size of the treatment group, the control group, etc.
Michael: Sure. So in the paper, we really describe just the second study we did, we actually did two studies, and in both studies we had almost 400 students. In the first study we had 395 students, and the second study we had 397 students. And some of the students were taking a critical thinking class from me, and the rest of the students were taking other philosophy classes. So in the study that we report on, we had 113 critical thinking students and 284 control students. Then what we did is, in the first week of class, we gave everyone a pretest, and this involved the five cheat codes, and then two good arguments, or at least promising arguments. They weren’t necessarily great, but they didn’t use any cheat codes. And the reason we included these, what we called skepticism checks, was just to check if all we were doing was creating skeptical students rather than students who could distinguish between promising arguments and cheat code arguments. Then, at the end of the semester, we gave them the same survey, same seven questions, and not surprisingly, in both of the studies, the training group students did much better than the control students. In the first study, we waited three months. So basically, the way to think about it is that these classes all happened in the fall. We waited till the end of the spring semester. It might have been four months, and gave them same questions again in the first study. We gave them the same questions again. The trained students did much better, and then we gave them different questions, and they still did much better. So in the second study we did, which is really the only study that we report on in the article, we gave them pre-test at the beginning of the semester, post-test. Then we waited a full academic year plus the spring, so we waited 16 months and only asked them different questions. So the pretest and posttest were the same questions, and again my students did much much better. But then we waited 16 months and asked them seven different questions, and the trained students were still doing much better, detecting the cheat codes and recognizing that the good arguments were still pretty good arguments.
John: DId the trained students do significantly better with all 5 of these cheat codes?
Michael: No, they did better on 4 of the 5 cheat codes. They did better on the correlation, regression, sunk cost, and opportunity cost cheat codes, but in all the studies that we did, on all the surveys that we asked, we just couldn’t get the trained students to do better on the gambler’s fallacy questions.
John: But still, having students do better on 4 out of those 5 cheat codes so long after the initial intervention is a really impressive result. One of the things you talk about near the end of the paper, though, is that you note that there are some possible sample selection issues. Could you just talk a little bit about what source of sample selection might have affected the magnitude of those results?
Michael: So one worry that one might have is that the students were not distributed randomly. So the students chose to take the critical thinking class, the students chose to take other philosophy classes, and so there might be something about the students who chose critical thinking classes that would incline them to be better critical thinkers, despite the fact that on the pre-tests, all the students in both groups did equally badly. So that’s one worry one might have, and the thing to do there is, although it’s not really easy given the institutions of academia, is to randomly distribute students into critical thinking classes and other classes. So that’s the main selection worry. Another reason why we might have gotten good results is that I had those students for 15 weeks. Some interventions talk about full- semester interventions, but often they’re fairly short interventions. I was able to have the students for 15 weeks and made them practice a lot.
Rebecca: It sounds like in the design of the experiment in your class, just generally, there’s lots of opportunities for feedback and correction throughout the semester.
Michael: Yes, absolutely. There’s a lot of feedback, a lot of correction, and I have a big bank of good and bad arguments that I can give them over the course of the semester that use these cheat codes or that don’t use the cheat codes. And one of the things I do for an essay in this class is ask the students to put themselves in the shoes of someone who’s trying to come up with a cheat code argument and ask them to give me bad arguments that use these cheat codes, and they actually turn out to be pretty good at it. And many of the examples that I use on later students are examples that prior students have come up with.
Rebecca: Sounds like it makes it a little bit fun too.
Michael: Yes, I do ask them for any humorous examples or examples that us boomers would have a hard time understanding. [LAUGHTER]
John: One of the things we appreciated about this study is that the data, the results, and the analyses that were done are all reported in more detail on the Open Science Framework website. We’ll include a link to that website in the show notes for this episode.
Rebecca: So we always wrap up by asking, “What’s next?
Michael: A few things are next. One is I’m writing a textbook that uses the materials that I teach. The way I teach critical thinking is different from the way other people teach it. I haven’t found a good textbook that teaches critical thinking the way I like to teach it. So I’m currently writing a textbook that hopefully will help other people get good results. I’m also planning to build, if I can, a game that anybody can play to see whether they can avoid being bamboozled [LAUGHTER] by cheat code arguments, this is sort of a longer term project because writing a book is going to be useful, hopefully, for people to teach this. But what’s really important, in my view, what makes this stick is practicing on lots of examples. And so, in a game where people see an argument and you say, “Well, is this promising argument, or is somebody trying to manipulate you?” we’ll hopefully give people practice at being able to detect those cheat code arguments. So those are two of the projects that I’m working on.
John: On the latter project, will it be a board game? Will it be an electronic game? Will it be an online game, or is that something that’ll be decided further in the future?
Michael: No, it’ll be an online game. Is what I’m hoping to do. I want to do something that can be used not just by my students and that’s easily accessible.
Rebecca: Sounds like some great projects. Thanks for joining us today.
Michael: Oh, you’re welcome. It’s been a pleasure.
John: Thank you. I was really impressed by this study [LAUGHTER] because, as I mentioned before, it’s so rare to see effects that are so significant and persistent. It’s, I think, something that people can learn from in terms of other types of issues where it’s hard to convey this sort of long-term impact on students.
Michael: Well, thank you so much.
[MUSIC]
John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.
Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.
[MUSIC]