Equity gaps in educational outcomes play a major role in perpetuating economic inequality. In this episode, Philip Oreopoulis joins us to discuss his research examining how tutoring and computer-aided instruction can be used to reduce disparities in educational outcomes. Philip is a Distinguished Professor of Economics and Public Policy at the University of Toronto, the Education co-chair of MIT’s Abdul Latif Jameel Poverty Action Lab, and an award-winning researcher who has conducted a wide variety of studies relating to education and educational policy.
Shownotes
- Nickow, A., Oreopoulos, P., & Quan, V. (2020). The impressive effects of tutoring on prek-12 learning: A systematic review and meta-analysis of the experimental evidence. July 6. NBER.
- Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4-16.
- Carlana, M., & La Ferrara, E. (2021). Apart but Connected: Online Tutoring and Student Outcomes during the COVID-19 Pandemic. EdWorkingPaper No. 21-350. Annenberg Institute for School Reform at Brown University.
- Deslauriers, L., McCarty, L. S., Miller, K., Callaghan, K., & Kestin, G. (2019). Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences, 116(39), 19251-19257.
- Khoaching with Khan Academy: Leveraging Computer Assisted Learning, Teachers, and Parents to Facilitate Personalized Learning at Scale. J-Pal
Transcript
John: Equity gaps in educational outcomes play a major role in perpetuating economic inequality. In this episode, we discuss research examining how tutoring and computer-aided instruction can be used to reduce disparities in educational outcomes.
[MUSIC]
John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.
Rebecca: This podcast series is hosted by John Kane, an economist…
John: …and Rebecca Mushtare, a graphic designer…
Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.
[MUSIC]
John: Our guest today is Philip Oreopoulis. Philip is a Distinguished Professor of Economics and Public Policy at the University of Toronto, the Education co-chair of MIT’s Abdul Latif Jameel Poverty Action Lab, and an award-winning researcher who has conducted a wide variety of studies relating to education and educational policy. Welcome, Philip.
Philip: Thanks so much for having me.
Rebecca: Today’s teas are:…Philip, are you drinking tea?
Philip: My tea is coffee. I love coffee. I once looked for a reason not to drink coffee, I couldn’t find one. I love my black coffee.
Rebecca: A true researcher at heart. [LAUGHTER]
John: And I am drinking a bing cherry black tea, a custom Tea Republic tea made for Harry & David.
Rebecca: And I have Irish breakfast tea. I really need to get some new tea [LAUGHTER]. I’m going to a tea store this weekend, so I’m looking forward to getting some new options.
John: And we have lots of tea in the office, some of which may not be as fresh as it was a year and a half ago. But this one still is good. It was purchased right before the shutdown.
Philip: You guys are inspiring me. I think I’m gonna have some tea sometime today.
Rebecca: All right, good, good.
John: In a November 2020 Scientific American article, you describe a meta analysis that you worked on with some colleagues that found that tutoring results in significant improvements in student learning. Could you describe this meta analysis a bit and what you found?
Philip: To backtrack a little bit, how it got started: my colleagues at J-PAL, Vincent Quan, Andre Nickow, and I, had heard about the potential of tutoring to be an effective form for increasing test score learning performance. For example, there’s Benjamin Bloom’s seminal article in the 80s, where he had two very small studies done by his students that both found off the charts improvement from offering tutoring in randomized control trials. In fact, that’s why he called it the ‘“2 sigma problem” that he found estimated impact from these two small studies were raising learning performances by enough to potentially solve most of our problems that we would be having in education policy. There were a number of recent studies as well, a randomized control trial coming out from the University of Chicago’s Ed lab, also finding very promising results from an RCT looking at providing in-class tutoring to grade nine students. And so we wanted to explore whether there was some consistency in these results, so we decided to try to take a more systematic look, and we gathered up all the RCTs, randomized control trials, in the last 40 years for about 96 studies, and we took a look and we found that consensus was quite remarkable. About 80% of those studies found significant effects larger than .2 of a standard deviation, and the average effect size was .38 of a standard deviation, which is like the equivalent of almost an entire extra year of school, from receiving these programs. And not only were the impacts really quite meaningful, as about as large as you get from education interventions, but they were consistent across the board. I think that this is about as much consistency as you’re ever going to get in an education policy intervention. So we were quite excited about that. We found that the effects were pretty consistent no matter which type of program that you looked at. They were larger for things like in-school delivery, three days a week, one-to-one delivery, full time tutors, but even in cases where that wasn’t the case, usually there were still significant effects.
Rebecca: Can you talk about what age the students were, what grades they were in?
Philip: It was for K-12.
John: I think it’s probably safe to assume, though, that the same effect would hold in the college environment as well. Those are some pretty dramatic effects.
Philip: Of course, to some extent, maybe it’s not that surprising. Giving instruction one-to-one leads to higher learning gains, and the biggest challenge, of course, is cost. We can’t all have our own teacher when we go to school. And so the biggest challenge, which gets back to Bloom’s point calling this the 2 sigma problem, is I think we have a powerful intervention to help education, it’s just that it costs too much to implement it on a larger scale. So the fundamental problem is to figure out a way to scale this in a way that can complement the classroom instruction.
John: And so that’s one of the things I think you’re looking at now, how this can be scaled up in a more cost effective manner. Could you tell us a little bit about your current research in terms of computer-assisted learning?
Philip: Sure. So computer-assisted learning or computer-assisted instruction is a type of educational software designed to help students progress through topics at their own pace. It has a lot of similar features as what you might receive when you’re receiving tutoring. So a typical example might be Khan Academy, MATHia, there’s lots of other types of software designed to help with different topics, math and reading, but they all have these sort of common features that allow students to progress through topics at their own pace. You receive immediate feedback from trying to work through your own problems and a chance to understand where you went wrong. If you do make a mistake, there’s data that’s generated from going through it that someone like a teacher might be able to follow and respond to. And so computer-assisted learning can, in some ways, simulate the tutoring experience, but of course, at a much lower cost. The challenge is you don’t have a real person guiding you through it. So even though a platform like Khan Academy is easily accessible, your willingness or motivation to go through it on your own is probably not as great as if you had a real person guiding you through the same material. So there has been some experimental evidence on computer-assisted learning, not as much as theories on tutoring, but of the 15 or 20 randomized control trials that have been done in this area, they have also been showing quite promising results. In cases where computer-assisted learning is provided, especially during a school setting, those receiving it also seemed to be performing at significantly higher rates than those in the comparison group. So there does seem to be some promise at using computer-assisted learning to generate the gains that we see from tutoring. But the way to introduce it, the instructions that teachers need to learn how to use it effectively, are not yet maybe as developed as we’d like them to be. So getting to, I guess jump into what I’m working on, I think that there’s a lot of potential for leveraging existing resources to combine with computer-assisted learning in a way that might come close to the tutoring experience. And so what I’m thinking of is in the classroom, that the kind of facilitated practice that might go on, say, in a math subject might be much better through a tool like Khan Academy than paper and pencil that we often give students. And so the question I’m investigating is around reshuffling the classroom in a way where the teacher is trained how to use computer-assisted learning more effectively in the classroom to generate that type of experience. So in the context of the program that I’m looking at now, which tries to integrate Khan Academy more into math classes, the teacher is still instructing and presenting topics, but now emphasizing the students following an individualized roadmap that allows the students to progress at their own pace, rather than having to keep up even if they’re missing on topics and not understanding. So the program which we’re calling “Coaching with Khan Academy,” or CWK, has students receive a roadmap of incremental topics and videos to follow at the start of school that roughly proceed in the same order that the teacher is going through. Now, the teacher has the students to try to work on this roadmap for at least an hour, an hour and a half a week, and tries to facilitate that time during the class and encourage more done at home, and the students then have the ability to hopefully get into a routine of watching a video and taking the exercises, and if they don’t score high enough on the exercises they’re asked to try to understand why they made the mistake using the hints and tips and guidance that Khan provides or gets help from the teacher, and then repeat it so that they don’t move on to the next topic until they’ve mastered that. So the students are not proceeding all in the same pace, but it is just a much better way to learn math such that the students don’t go on to the next topic until they’ve established a strong enough foundation on the first one.
John: During the global pandemic, most high schools moved to emergency remote instruction for an extended period, and there’s quite a bit of evidence that that led to a decline in overall learning, but also some growing achievement gaps which are tied to household wealth and the wealth of the school districts in which the students reside. What types of policies could be implemented at the K-12 level so that students are more equally prepared for entry into college
Philip: On COVID, we’ve all been exposed to online learning now, and most research suggests that it’s not a great substitute for in-person but there are certain benefits from being able to speak with a real person over a computer in regards to tutoring. So the biggest one is convenience, both for the tutor and the tutee. It’s nice to be able to jump in on a call and spend just 30 minutes on that or an hour, and not have to drive to the person’s location or do this after school. The opportunity to facilitate more tutoring, I think, is increased by having this online access. So I think there’s a lot of interesting promises from that. This one particularly interesting study that was done during COVID last summer, where a group of Italian faculty organized a volunteer tutoring experiment where they got the Deans of their respective universities to invite university students to volunteer their time, three to six hours a week to reach out and connect with students who have been struggling in the high schools and lower grades. And on the flip side, they got the school districts of several locations in Italy to ask teachers to identify students that they thought could benefit from having this one-on-one instruction. And then the response was great in both ways, there were a lot of people willing to volunteer their time for this effort, and there was also a lot of perceived need for students that needed this. And so from this large set-up, they randomized who they were able to give this offer of assistance to. And it was done all online, sometimes over the phone, but more often through Zoom, or Skype, or whatever was most convenient for the match to take place. The tutors met with tutees, for three hours a week, over six weeks. The topics were either math, Italian, or English, and then at the end, the researchers collected the survey and found similar gains to what we were finding in the online overall. Not only that, but they also collected data on mental health and found improvements in feelings of connection, more positive outlook on life. And what’s also interesting as they seem to show improvements and positive outcomes for the tutors themselves, as well. So it stands the potential for a win-win, and this was all done online. So it’s like the only online study I know, but it seems to show the potential that it might be done there. One other example I should mention is Khan Academy has also initiated another organization that facilitates free volunteer online tutoring. It’s called ‘schoolhouse.world’ and it’s been interesting to watch that trying to get up and running. Their system allows anyone in the world to volunteer their time as a tutor, and then they try to connect anyone in the world wanting to receive that tutoring. And you get some sense of some of the challenges from doing that. How do you screen for quality? And also, how do you screen for safety? So they’ve had to go away from a one to one model to more of a group model. They’ve had to have systems in place to check the quality of the tutoring, what’s being discussed. They’ve had to switch to allowing only high school students to receive the tutoring and a few other challenges. And so there’s challenges but also a lot of potential in this that wasn’t available from always having to meet your tutor in school or after school or face-to-face. So the potential scalability is enormous, and that’s where the intriguing possibilities are with that tool.
Rebecca: So if we’re looking to reduce achievement gaps, we’ve talked a little bit about COVID and the mix of instruction that students might’ve had during COVID, the quality of instruction, access to technology, to even have interactions with teachers in some cases, and historically even, differences in ability when students arrive in higher ed. What are some of the things that the higher ed community might be thinking about in terms of this research? Should we be advocating for certain kinds of policies or programs in K-12? Should we be trying to institute some of these things in higher ed? What are your thoughts on that?
Philip: So just in terms of advocacy and thinking about facilitating more equality, there’s no question that tutoring has, in general, been an unequal program. There’s the whole private sector of tutoring where a lot of households for more affluent families seem to receive it than those from less affluent households. And so one thing we can do as policy-makers is to try to facilitate more tutoring to happen in schools, especially at schools for more disadvantaged backgrounds. We can also focus on providing tutoring to those who need it most. I think that there is a growing awareness of the potential for tutoring to make a real difference in helping address the learning loss that may have occurred with the pandemic and just helping address education inequalities in general. And so a lot of resources have started going towards trying to increase the amount of tutoring happening in schools. I think that the more we understand how to implement it successfully, the more guidance that we can provide the K-12 sector in trying to introduce that. I think that there is a lot of optimism now around its potential. I think tutoring is one of the most effective programs that we can offer to make a meaningful difference at scale, such that we can get more students arriving into post-secondary ready to handle it and succeed well there. So that’s on that end. I think that there’s no reason why we also can’t consider tutoring at the post-secondary level as well, and the potential benefits that might come from that. Even if we just look at first-year calculus, or other subjects in math, computer-assisted learning is well developed even at that level, the need for tutoring at that level is there as well. And so it really does go from that importance of establishing a foundation that one might benefit from tutoring at earlier ages. But even at the post-secondary level, regardless of what level the student is, we can all benefit from one-on-one instruction compared to being in a calculus class of 500, right? I think there has been less research that’s been done in that area, but the evidence certainly points to the direction that tutoring at the post-secondary level would be also effective and important to consider.
John: And you mentioned that Italian experiment where college students were providing tutoring, and you mentioned that that was a very positive experience for the college students as well. That might be an interesting model where college students could improve their own skills and develop a bit more automaticity and more practice in basic concepts, while helping bring students up to a higher level in secondary schools. That’s a program that I think offers a lot of potential.
Philip: So I would agree, absolutely, the expression is you don’t really understand something until you teach it. I think that there’s something to be said for that. I think that there’s also a lot of skills and experience that is gained from trying to help others, from trying to connect with perhaps younger individuals that have not had the same background as you. I think that the experience is also attractive to employers looking at who to hire. I think there’s huge gains from all the things that you might volunteer or use your time for in college, spending some time to volunteer to do something like tutoring could be a very rewarding thing as well. So I’m also excited about that model. I think that there are ways to try to facilitate that kind of model at scale and more research needs to be done to explore how to do that.
Rebecca: One of the things that I heard you mentioned early on in the conversation is the idea that, historically, folks who had access to tutoring are more affluent. So the students who most need the tutoring are the ones that aren’t always getting it, because they can’t afford it. So I love the idea of having it in schools or it’s a part of our programs. But also I think sometimes tutoring has a negative connotation to it. It’s like a deficit model. Especially I’ve seen this in higher ed, students don’t want to go to a tutor because it makes them feel like they’re dumb or something.
Philip: My first reaction to that is that tutoring can be beneficial at any level. For example, in the Khoaching with Khan project that I’m looking at, the potential is to help all students in the class regardless of their level, because every student can be given their own individual roadmap. And that not only includes those that are behind grade level that benefit from establishing a stronger foundation in that earlier material so that they can catch up, it also includes those at a higher level that don’t have to be held back or wait for the instructor to cover new material can use a platform like Khan Academy or a tutor to work on more challenging material that interest them. And so how to remove that stigma that exists in general, I agree the usual perception is when someone asks, “Do you need a tutor?” it’s because you’re struggling. It doesn’t need to be that way, but at the same time, I think the more we become aware of the benefits from the tutoring, the more we realize that it’s a great resource to take advantage of. Getting back at the college level, I don’t know about your own experiences, but it always amazes me how few students take advantage of all the free tutoring that’s being offered by the universities through, like, office hours. The opportunity for receiving one-on-one discussion is often there, and yet so few students seem to take advantage of it, perhaps because of that stigma or perhaps they’re too busy. Some of us, when we went through college, were pleasantly surprised by how much you can get with office hours of graduate students and extra tutoring and how much you can learn from that process.
John: As in a lot of classes, students are treated as if one-size-fits-all education and students come in, especially in subjects such as math where there is a very rigid structure, if you don’t have a solid foundation and concepts, learning new topics is not going to be very productive, because you don’t have that foundation to connect to. And I see that in my own classes, and it’s a bit of a challenge to try to do that. Because of issues of scale I often teach large classes, I try to rely on peer instruction as much as possible with small group activities. Could small group peer interactions in working through problems and problem sets achieve something similar to the one on one attention?
Philip: In the literature, it’s called peer-to-peer, we did not look at peer-to-peer in our meta analysis on tutoring, but there is some literature and there’s some effort to consider that. It’s a little bit of a different model, because you’re relying on slightly older students or similar students to help assist other students. I think more research needs to be done on how to make that happen effectively. On one hand, the potential is there to make this a scalable, effective program that doesn’t cost very much. On the other hand, monitoring quality and the potential to train to be a tutor and to do a good job with it may not be there as much as with the regular type of tutoring program.
John: In particular, I was thinking of activities in class where students work on problems in groups, and they try to argue out solutions. They work together and they can explain to each other things they don’t understand, but the key aspect of that is they get feedback on whether they’re correct or not, some constructive feedback on where they went astray. But I was just thinking that those types of small group interactions could provide some of the benefits without that stigma of needing to go to tutoring and perhaps at a higher scale than tutoring might work.
Philip: The advice that I often give my students is to study until you feel you can explain it to someone else. And so there’s a similar, perhaps, mechanism at play when we’re thinking about that. When you try to write down a concept or explain it, even to yourself, out loud or to someone else, you quickly realize what you understand and what you don’t. There does seem to be a lot of potential there.
Rebecca: Sounds like one of the keys to reducing stigma around all of this is making the coaching or this tutoring model just something that’s normalized. Maybe it’s normalized in class, it’s normalized through the school day, and then people might be more apt to take advantage of it because they have access to it. But also, it becomes a standard way of being, that’s what other people around them are also doing.
Philip: Absolutely! I think if we can reframe tutoring as just individualized instruction or personalized instruction, then we can all understand the potential benefits of receiving more personal help than in a classroom setting, and that goes for pretty much anyone.
Rebecca: It really also matches up well with a lot of universal design for learning principles of flexibility as well, and allowing students to go at their own pace and finding ways of teaching and learning that match well for students and where they’re at.
Philip: And of course, the issue is scale. Getting children to learn in a classroom of 25 to 30 students, when these students vary enormously in academic levels, is just really difficult. And trying to figure out a way to provide that individual attention is the challenge that all teachers face and have been facing for many, many years. And if we can find a way to scale adding on or providing more and more individualized attention, it has the potential, I think, to make a real difference in education. Of all the potential policies that we can be looking at, I do think that, at the school level, leaning towards more individualized instruction is where we should be looking at, for a solution.
Rebecca: It’s so interesting to me that we’re having this conversation early on in our semester, because after teaching online for a year, which I hadn’t done previously, I’ve really worked to make my classes more flexible and actually offer some of those kinds of models that you’re describing where students are going more at their own pace, and that they can get some individualized instruction when they need it and that they need to do this mastery learning so that they build on things over time. It looks to me like maybe I need to look more into tutoring and coaching models that have worked really well to see if I can’t implement some of that more during class time.
Philip: There may be different ways to do it. Some may be more effective than others, but I do think, getting back at what John was saying, it’s harder to provide that individual support or help to students arriving in college without that foundation. I have done some other work at the college level, trying to facilitate more personal attention to students arriving, trying to help them out and encourage them to get into better habits, and it has proved quite difficult to change behavior, and so I have found myself reacting to that by focusing more on earlier grades to see if there might be more promise on trying to foster better study habits, better learning habits, earlier on with the hope that students arrive in college more prepared.
John: I think that’s one of the things a lot of behavioral economic studies have found. Interventions that result in long-term changes of behavior are challenging in general.
Philip: Absolutely.
John: And I think you’ve done some research on that.
Philip: Absolutely. So if we have to change one-time actions, like helping students through applying for college, applying for financial aid, those types of interventions are much more promising at affecting one-time goals than to change habits or routines that involve much more continuous behavior. So helping someone study more effectively, spend more time studying, these are much harder problems to solve. And maybe low-cost nudges that we’ve been looking at in the literature may not be as effective. I think that does tie back into how my perspective has changed over time. It’s hard to have significant influence without personal connection. It’s a lot more expensive, but there’s only so far you can go with sending an email or a text message or a one-time meeting in trying to change someone’s learning trajectory or life trajectory. And the more you sort of look at education policies that have been successful, the more you notice that they often come with this personal connection that’s been important for making that meaningful change.
Rebecca: It seems like we should all be really advocating then for these much more early interventions. It’s much more cost effective if we get those habits in place really early [LAUGHTER].
Philip: I will say there’s surprisingly not enough research on the long-term effects of tutoring. I’ve seen one study that has found that the benefits of receiving that tutoring continued one year past the program ended; the effects faded, but not by that much, and that’s the only study I’m aware of that actually does a long-term study. So on the question of whether we can have these life-changing impacts from targeting earlier ages, certainly, there’s a literature for the very young… like, almost helping at the household, but at the school, I think that more work could be done.
John: And that could be a really productive research area. Before we started recording, we were talking a little bit about, with the pandemic, creating our own videos. Could you talk a little bit about how you try to implement what you’ve learned in your own classes at the college level?
Philip: Yeah, I think that using the situation last year to put my lectures online has freed up space in the actual lectures to be more interactive. So I think it was a benefit both ways. The videos of the lectures themselves became more streamlined, I got a chance to break them up into smaller parts, sort of like Khan Academy videos, where instead of one video that’s two hours long, that goes all over the place, and you’re staring at me and the Blackboard, I created five- to ten-minute videos of vignettes that I could focus on with slides and have a series of these videos that students could watch at their own pace. I could edit them and make sure that the video is as succinct as possible and gets across what I really want to say. So that was good on the video side, and then on the actual lecture side, we spent that time going through problem sets and answering questions and it was much more interactive, closer to the spirit of more personalized instruction. So there was more opportunity for questions, more opportunities for the students to get more involved, and I think it did lead to more satisfaction of that approach. Obviously, the big question is, ‘Do they really watch the videos when they’re asked to do it on their own?’ I think there are ways to try to incentivize that, but just like any class, the students really perk up when they’re working on a problem that was, say, a previous exam question.
John: I’ve used a very similar approach. I’ve used videos for like 20 some years in my classes, but one thing I started doing last year is I embedded questions in the middle of the videos, and that’s a pretty effective incentive structure. It does get them all watching the videos, and at least thinking about it and trying to make some connections while they do it, and that’s worked pretty well.
Philip: Not only that, but you can make them mandatory for class participation. So you stick those questions in and they have to watch the video to find the questions when they pop up, there’s software that can do that. And then you can make it as a way to encourage them to have to watch the video.
John: Do you think that more use of computer-aided instruction is going to be helpful in allowing more students to be successful?
Philip: I’m very optimistic on this potential of leveraging computers with teachers and parents working together on trying to facilitate high-dosage practice. We’ve been talking mostly in math, but it could also be language as well, and maybe other topics. But I think this really is a good way to learn, as long as the practice time is long enough, and the student’s not stuck. I think that it takes a while to get into the habit, getting used to the software, getting used to the routine, both for the teacher providing this and for the student doing it, and so that, for me, right now, is the biggest challenge. I am optimistic that if we can facilitate a way to help teachers and students get to that higher-dose practice using computers, then very good things will happen. I think that the evidence is highly suggestive that the high dosage is a worthwhile thing to get done. I’m hoping that we can generate evidence that that’s the case, but we are finding that there are challenges because there’s a learning curve, it is changing the way that the classroom is done and changing the way the student usually learns, but I’m optimistic that if we can get past that, the students and the teachers will come to like this approach, and that we can do more of it at scale.
John: And I think a lot of people began experimenting with some sort of a flipped approach where they created videos and then use the classroom for more interactive activities, ast least at the college level, I don’t think that’s happened quite as much at the secondary school level. But I think that has helped provide at least some professional development for faculty. But it is an adjustment that students are not adjusting to perhaps as easily as I would like, I know I always have trouble getting across to students that there is some benefit of working through problems in class and watching videos and learning some of the basic concepts outside of class. Students would rather be lectured to, there was that big study that was done at Harvard not too long ago, where students were asked about active learning classes versus lecture classes, and the research certainly showed that active learning in the classroom led to significant learning gains, but students perceived a higher learning gain from lecture classes, and that’s where I think that issue of students’ adjustment is a challenge, and until we get to see a large amount of this occurring, it’s going to be a while convincing students of this, because it’s really easy to sit there in a lecture and nod and smile and have it all make sense and it seems to fit together very logically, but then when you try to apply it, there’s a bit of a problem, and then the questions are somehow unfair. But when students are faced with problems and interactive work in class, they’re confronted by not knowing things as well as perhaps they thought they did, and it’s not as pleasant of an experience. And I think that’s the source of that metacognition, that students perceive that lectures are more effective, because it’s easy to sit there and listen in, and it all seems reasonable. But the problem is when they try to work through problems and realize they don’t quite have those connections fully there yet.
Philip: The lecture seems to make so much sense until you sit down when you get home and try to go over it again, but I do think there’s the potential for this middle ground that even in the experiment we’re looking at, we’re not entirely flipping the class, in fact, we want to work with the teacher to understand what their own preferences are, while still trying to hit this high dosage of practice, which may occur in class, but also could occur at home as well. And I think that there is something to be said by having a lecture of a new topic being done in class, in person, with the real person. It gets back to that importance of personal connection that the computer is not able to provide. And so maybe there is a sweet spot around providing real instruction, real empathy, but also enough time to be working through these problems at your own pace. My vision for the Khan project is that students say, in grade four, getting 90 minutes of math a day, maybe half an hour of that would be the teacher’s own instruction of a new topic, but then a lot of the other time would be students working on their own devices, while the teacher takes the time… instead of just sitting up at their desk… walks around and spends a lot of time looking over the student’s shoulder, using the data that they’re seeing to understand who’s struggling and where, and spends a lot of time working individually while the student is using the computer. So there’s still that interaction going on and taking advantage of the personalization. I think they too can go really well together.
Rebecca: That’s definitely something I’ve been experimenting with. I went all the way flipped before, and right now I think I’m right in the middle. There’s some flipped, there’s some demos that are live so that people can interact and ask questions, and then there’s lots of practice with individualized attention. And it does take a little time to get everyone on board, to get everyone trained to do things in a new way. So in a 15-week semester, it might take two full weeks to develop new habits and workflows for everyone, but really after we get over that two- week hurdle at the beginning of the semester, my classes tend to settle into a routine that seems really productive and that students have been pretty positive about.
Philip: A key feature of the coaching with Khan program, is that every teacher gets their own coach that we spell with a “kh,” and our coaches meet with the teacher prior to school to go over our suggested recipe to follow, but then they don’t just leave it at that, they keep working with the teachers to check in and try to troubleshoot or brainstorm or reassure and remind the teacher until things are going smoothly. But it can take longer than two weeks to figure out how things are going, and then on the student side, it can take a while for them to adapt and understand that there’s some independence on their own for wanting to do it. The hope is that the students start to gain confidence when they see their own progress, when they see that maybe they didn’t consider themselves a strong math student, but if you start them at the right spot on this roadmap, and then they proceed incrementally, and they can see that they are advancing, then they start to understand the potential benefits and internalize the desire to keep going on their own.
Rebecca: Yeah, that autonomy and that empowerment, I think, is really key to the whole puzzle. And I think something that probably tutoring historically helps students achieve is that they can do this. They might have a little extra guidance initially, but then they achieve it and can do it, and that’s really empowering.
John: That’s our hope
Rebecca: We always wrap up by asking: “What’s next?”
Philip: What’s next? I think I made some notes on that. [LAUGHTER] So I think the issue around tutoring and individualized learning is all about, now, scale. I don’t think we need another study to demonstrate that one-on-one instruction, or one-on-two is an effective additional tool for learning, that more should be done if it were possible. A lot of resources are now going into trying to provide individualized instruction. I think a lot of policymakers and governments are looking to tutoring as a way to address some of the learning loss that may have gone on during the pandemic, and I think, in that space, there’s some optimism by researchers and policymakers to try to understand what types of scale up are better than others in a way that we can make a meaningful difference at the aggregate level.
Rebecca: Well, thanks so much. I’m really excited to hear more as your research develops and more information becomes available!
Philip: It was a pleasure to get a chance to chat with you guys. It’s a topic I’ve been spending a lot of time on and losing a bit of sleep on trying to get things to work. The experiment that we have going on, this is going on in Texas, and one of the challenges of doing a field experiment is that so many things go wrong while you’re trying to deal with real people, real students, and provide evidence that this is a good idea. And it’s always a bit frustrating to face these challenges, like just account issues, students have trouble getting on to Khan Academy and the teachers getting frustrated, and it would be a shame to have those issues that can be worked out actually create this wedge from the program going smoothly and making the difference between having these great impacts or not. So it is stressful, but I think it’s worth it to try to keep at it, and I hope to be able to do so. With funding and policy support we’ll just keep trying. I think there’s a lot of interest in it, I think that it hasn’t been difficult to motivate these ideas and wanting to do more on it. So thanks a lot for giving me the chance to share these thoughts.
John: Your work is incredibly important. And so much income inequality is associated with differences in educational attainment, that understanding these achievement gaps and what we can do to narrow them can have a really dramatic impact on society.
Philip: Fingers crossed!
[MUSIC]
John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.
Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.
John: Editing assistance provided by Anna Croyle.
[MUSIC]