257. PsycLearn

Adaptive learning platforms provide each student with a customized learning path based on the student’s individual learning needs. In this episode, Anna Yocom, Linda Goldberg, and Alan Strathman join us to discuss how the American Psychological Association has developed adaptive learning packages for core psychology courses.

Show Notes

Transcript

John: Adaptive learning platforms provide each student with a customized learning path based on the student’s individual learning needs. In this episode, we examine how one professional association has developed adaptive learning packages for core courses in their discipline.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are all involved with a psych learning project, which we’ll be talking about throughout the podcast. Let’s have each guest introduce themselves.

Anna: Hi, I am Anna Yocom. I am a senior lecturer at The Ohio State University in psychology and I am also a Content Manager for PsycLearn.

Linda: And I’m Linda Goldberg. And my role with the PsycLearn project is that of product evangelist.

Alan: Hi, this is Alan Strathman. I’m a content manager for PsycLearn and was a professor of psychology for a long time at the University of Missouri.

John: Welcome, everyone. We’re really happy to talk to you about PsycLearn. I first heard about this when I was working on a project at SUNY dealing with adaptive learning solutions. And we had seen some write ups of PsycLearn so we want to find out a little bit more.

Rebecca: Today’s teas are:… Alan, are you drinking tea?

Alan: I’m drinking coffee.

Rebecca: Oh, a coffee drinker.

Alan: Sorry.

Rebecca: I see how it is. [LAUGHTER]

Alan: I had tea this morning this morning.

Rebecca: Oh, that’s good. That’s good.

Rebecca:: How about you, Linda?

Linda: I’m having an iced tea this afternoon. It’s hot and muggy outside, so something citrusy and decaf.

Rebecca: …and perfectly refreshing. [LAUGHTER]

Linda: That’s right. And sweetened.

Rebecca: Awesome. Anna?

Anna: I had tea earlier, but I have switched to water.

John: And I’m drinking spring cherry green tea.

Rebecca: Nice. I have some Scottish breakfast. Still in my pot today.

John: We’ve invited all of you here today to discuss the PsycLearn adaptive learning platform developed by the American Psychological Association. Could you tell us a little bit about the origin of this project?

Linda: Okay, I’m going to start with that. And there are a couple of overlapping facets to the origin story, so to speak. One is really a desire to expand APA’s publishing program. APA has a very rich publishing program, which includes publishing of over 90 academic journals, a very robust books publishing program, where many of those books are aimed at clinical practice, as well as books that might support the professional researcher, books that might support self help, and so on. And we found that there was an opening for us to make a contribution to undergraduate academic instruction and contribute to that portion of the path toward the profession. We also were recognizing as we contemplated this new project that there was a tremendous amount of disruption in the higher educational publishing landscape, a lot of cost pushback, a lot of desire among publishers to leverage technology. And we thought that perhaps not simply for delivery of content, but to improve the learning experience and to improve efficacy that we saw there might be some opportunity there.

Alan: Yeah, also, let me jump in. I think the developers also saw this as an opportunity to work toward some of the goals that APA has for the field of psychology, including, for example, utilizing psychology to make an impact on social issues.

Linda: And the field of cognitive psychology is one that tells us a lot about how we learn. So we felt like we could draw from our very own field in development of our final product,

John: Which courses have been developed for this platform.

Linda: So we started with a course for research methods and psychology and added a PsycLearn statistics for the behavioral sciences to that. These two courses are typically required for all majors in the discipline. And they also represent the most challenging content for many students. So we thought that seemed like a good place to start.

Rebecca: Can you talk a little bit about how this material was developed?

Alan: Sure, we use a backwards design process where we start by identifying what we want students to learn in each module and a module is similar to what would be a chapter in a regular textbook. These outcomes that we have in mind become Learning Objectives, and they really guide the content development. Learning Objectives are developed through collaboration, a conversation, maybe a hashing out between the subject matter experts and the content managers, and are influenced by a variety of factors, including how well the learning objectives address APAs guidelines for undergraduate education. And then there is a rigorous review process where other experts in the field provide feedback and often suggest additional learning objectives

John: Was there much of a challenge getting agreement on learning objectives. I know in many disciplines, there’s a lot of discussion of what should be in the core material in the discipline? Has psychology been able to overcome that a bit by agreeing on these standards?

Alan: Well, I think it is a challenge in every case, because everybody has a sense of what they think should be taught and what is essential for students to learn. Like I said, I think it’s hashing out. It’s sometimes a little bit of trial and error, where we identify a learning objective, and then start writing about it. And then at some point, we may decide that’s not the right objective to have. But I think in the end, the subject matter experts and the content managers do a really good job of coming to some consensus on what we want to teach.

John: Was the content in the courses, the actual readings and other materials, developed by the people who created the package?

Alan: Absolutely. We are a digital first product, with original content developed by experts in various sub disciplines in psychology. And this really gives us a big advantage I think over print textbooks. So starting from scratch, in an entirely digital environment, we can develop a seamless presentation, where we start with learning objectives, we present content, we design activities and interactive exercises, and then have assessments. And we can be sure that all those things are aligned with learning objectives. And we don’t have to try and retrofit learning objectives into existing content. And then just hope that assessments that we have address the new learning objectives. So I think being able to work in a digital environment allows us to have this really seamless presentation where we start with learning objectives and we go all the way through assessment, and all those things match.

Rebecca: You’ve hinted a little bit about maintaining the content, given that it’s in a digital platform. But can you talk a little bit more about how you maintain that or what that review processes is or how that gets updated on a regular basis?

Alan: Well, that’s a good question. One issue for us is that we’re still so new, that we haven’t had semesters and semesters of feedback from students that we can use to make revisions. But what is really exciting is that whenever we decide that we want to make a change, we can do it in real time. Think about a print textbook, whenever they decide that they want something to be different, they might wait three to five years for the opportunity to make those changes. With us, since it’s digital first, we decide something needs to be changed or updated or revised and we can make those changes in a really timely fashion. And I think it really helps to keep our content current.

John: And if you find that students are not achieving the learning objectives, you have the ability, at least, to make some changes, either redefining the objectives if perhaps the objectives didn’t work too well. Or if the learning materials weren’t quite aligning with that. How often is that type of change done?

Alan: Well, so far, we’re still into the development process, that we’re making those changes just all the time. As the products get older and more mature, I think we’ll need to do that less. But again, we’ll still have the opportunity to do it as often as we’d like to.

Rebecca: John mentioned at the beginning of the episode about his interest in this project related to adaptive learning. Can you talk a little bit about the adaptive learning component of this project and the platform that you use.

Anna: The platform that we use is CogBooks, it is a platform, I think, is really unique compared to other textbooks, because it allows students to engage in their learning, and really direct their own learning pathway. And that’s based on their understanding of the content. So instead of going through a whole chapter, and then maybe quizzing yourself or testing yourself, after each content page, they get some formative questions and they get to make their own assessment. They get to say, “Okay, did I understand that? Was I not too sure? How did I do on those questions?” And then they can choose whether they want to get that support material or not, which we think of as like, “Would you follow up and ask your instructor a question.” That’s how we think of this support material. So if a student is very confident with the material, they’re essentially going to get a different path than a student who might need help in a couple of different areas or a student who might need help in a lot of areas. There’s just more examples or more practice. So it can look a little bit different for each student, which is pretty cool.

John: Linda mentioned earlier that psychologists have done a lot of research and cognitive scientists have done a lot of research on how people learn, how have some of those principles of how we learn been built into this platform?

Alan: Well, Linda is certainly right. There is a lot of research on learning science. And because we are psychologists, we’re familiar with the research, and use that knowledge to help students learn throughout the product. I think the centerpiece of our efforts to use learning science is the inclusion of metacognition, that is helping students evaluate their own learning, helping them identify what they know and what they don’t know. And when they identify what they don’t know, then they could spend a little more time learning that particular topic. Often students get to the exam, and they haven’t really done a good job of testing themselves, identifying what they know, and what they don’t know. And the first time they realize they don’t know something is when they get to the exam. And that’s obviously not the right time to get there. And so we’re able to identify ways that we can help them evaluate their own learning. We help them in this process by incorporating those learning strategies that we know are supported by research. We present concrete examples throughout. We have frequent activities and assessments. Students get practice retrieving content from memory. And these opportunities are spaced just as the research suggests they should be. They have interactive exercises. They get practice in the elaboration, or explaining content to themselves in their own words. And really the process of taking material and from the jargon in which it’s presented to explaining things in their own words, is a really useful process for students to engage in. And we could have a whole podcast just on principles of learning science that PsycLearn incorporates.

Rebecca: Can you talk a little bit about what the student experience is like for students that are using the platform? So Anna, you hinted a little bit at this with some of the adaptive learning components. But what does it look like from: I entered this space from the beginning, and now I’m interacting with this over time.

Anna: So students really, I think, approach PsycLearn differently than a printed textbook, again, not only just the fact that they would go through a printed textbook, and then maybe ask themselves some questions, probably skip over a lot of reading, because maybe it’s not very engaging or easy to get distracted with print textbooks as well. But one of the really interesting things is that they’re engaged with the content pages within them. So they’re not just engaged outside of content, like “Oh, refer to this video,” or “you can reference this activity” or “try some completely different platform to help with your reading….” but within the content pages, they get activities in which they’re asked to reflect on the course material. How does this connect to things you already know? So they get to respond to those things within the content pages. They get practice formative questions, formative meaning that it’s not just did I get this right? Yes or no. But they get feedback. So they would find out what a suggested answer was to a short answer question. They can compare that to their own. With a multiple choice question, they get follow up on why that is the correct answer, not just “Well, if I’ve missed this a couple of times, and they get multiple chances to respond, but they get some explanation about why that was the correct answer.” And so we have video questions, we have matching exercises, we have short answer, we have multiple choice questions, different drag and drop. So we’ve tried to just vary the kinds of interactions that students would have, the ways that they can practice answering questions. And this variety really helps maintain interest throughout. So they get to engage it in lots of different ways. It’s not just a rote memorization, but they’re actually applying the material as they go through each module.

John: So you mentioned formative questions. Are there are some assessment tools built in to evaluate students’ learning for grading purposes, for example?

Anna: There are. Yeah, so there are summative questions. And we try to have them associated with those learning objectives that Alan had talked about earlier. And so after maybe one, two, or a small group of those learning objectives are covered, there is a short sample, maybe 5…10 multiple choice questions where students have a few attempts, and we want them to get to about an 80% accuracy. And one of the nice things is if they don’t on their first attempt, they can actually go access those support pages then. So maybe they skipped over them earlier. And they thought, “Okay, I got this, it’s fine. I understood this page.” And then they get to these questions later on, these mastering the content and they’re like, “Well, wait a minute, I was missing some of this.” So they get another chance to go back to those support questions, and try it again. So then hopefully, they can get those right after three attempts. We don’t want them to be punitive. We want them to get engaged with the material and get to that mastery.

Rebecca: So now that you’ve talked a little bit about the student experience, can you talk a little bit about how this supports instructors? What kind of feedback is provided to instructors and maybe how instructors are using this in their teaching.

Anna: One of the things I really like, we all know, anytime that we can even maybe think back to our own experiences in classrooms, it’s a struggle getting students to read. But a lot of these classes, I mean, really all of our classes, if we’re assigning some sort of book, some reading material, we want them to read it. So how do we get them to engage? Well, that’s where the accountability comes in. So we’re giving them this tool, and then their instructor can see, “Okay, they’ve made it through this material, they engage with this.” So the instructor could choose to then reiterate that material in class, or they could choose to maybe take an active learning approach in class. And okay, now that you’ve had this background, everybody made it through, we’re going to apply it. And maybe not everybody got to 100% on every question, but maybe we got to about over 90% completion, which is what I would typically see in my classes. So you would have students who, even after a few attempts, would sometimes miss some of those mastering questions, but generally, it was pretty high. And so I could see that before I went into class. Okay, so students, as a whole, on average have got this. So I can take it a step further, where I can go back to those questions that were really challenging. So you get to track progress through each module, you can also dive a little deeper with individual students. So a student comes to you and says, “I’m really struggling,” the instructor can go in there, they can look at their pathway. Were they accessing the support material? Did they take those quizzes a few times? Where were they spending their time? Or if the instructor wanted them to cover it on let’s say, Tuesday night, were they doing it all Tuesday at 9pm? Right? Like you could actually see all of that. So you can kind of work with them like, “Well, I see you started this that night, let’s think about how we could maybe back that up. Like maybe we should pace this out at least a few days ahead of time, which is a really nice way to work with students.” And it also syncs with the LMS, the learning management system, so it really takes that burden off of instructors as well. So it’s talking to the platform that we get there. So we get this easy back and forth, which is really nice. So we can nudge students along with that, I would often encourage students to visit those support pages if they were struggling, like, “Look, I know this is really challenging,” or, “Hey, there’s more practice, if you just want a little bit more practice, you can go to some of those support pages to get that.” And the other nice thing is students can actually email right from there, which I think is pretty unique. So if a student emails me, I could see okay, they were having problems in statistics with variability and standard deviation. And it will take me as the instructor right to that page. And so then I can see that and I can say, “Oh, well, why don’t you try these examples here,” which was really nice. Instead of just a student saying, “I don’t know, I don’t get this,” they can let you know exactly where they’re struggling. So it provides a lot of options for instructors.

John: And do instructors get aggregate statistics on what areas students are struggling in addition to data on individual students?

Anna: Absolutely. So you can see that on the dashboard. You can see if there were chunks of mastering questions, for example, or a specific question that students struggled with. So you could choose to review that and maybe even go over some of those in class or go over specific topics in more detail.

John: How are instructors combining this with other face-to-face learning activities, for example, in face-to-face classes. As I understand it, most people use this as a textbook replacement, except a much more powerful replacement to a textbook. What other things will people often do to provide more of a sense of community or a social component to the learning?

Anna: Yeah, so, I think a lot of instructors are using it, just as you said, as a textbook replacement. And that’s how I have used it for statistics in particular. So it was taking the place of my other textbook, which actually was a print textbook that was online. And so I don’t feel like there were any losses. But we were gaining the ability to say, “Okay, we’re all on the same page. I know that everybody’s at least had some exposure to this. So now let’s talk about it. I can see where you were struggling. So I know we’re going to need more examples here. I know we’re going to need more follow up here.” Or “why was this question challenging? Why was this hard?” So I think drawing on those parts are how we can maintain those social ties and the social interactions we want whether it’s online, whether it’s face to face, or in any capacity.

Linda: I’ll just go ahead and point out as well that part of the content package that we’re making available to instructors who adopt, PsycLearn course material is a set of student activities that sort of exist outside of the platform itself and are available for instructors to assign or to bring to the class. They’re typically designed again to align with specific learning objectives and to give the instructor options for an in- class activity or an additional assignment activity. And that’s accessible to the instructor when they’re using the dashboard.

Rebecca: Sounds like some really great support materials. Can you talk a little bit about how this platform has affected equity gaps that we might be seeing?

Alan: Sure, I think there are two ways that PsycLearn is addressing issues of equity. First, we are designing PsycLearn to be culturally inclusive, that is we work to make sure that through examples, images, photos, figures, every student can see themselves in the product. Plus, our goal is to include the research of diverse psychologists and psychologists of color. And then I think the second way that we strive to be equitable is to design for universal access. We want the product to be accessible to all students, and so we make great efforts to do so. We use typography that’s particularly accessible. We incorporate transcripts and closed captioning, we have text-based alternatives for non text elements. And our efforts are well beyond what the minimum guidelines would be, well beyond what typical expectations would be, and I want to say that this is because we value this process so highly. We are keenly interested in making an accessible, inclusive product, and I think we’re really working hard to do that.

John: Have there been any studies of the overall efficacy of this program compared to alternatives?

Linda: Well, we have conducted some impact surveys, surveys of student users to gauge their impressions of the impact of PsycLearn on certain important aspects of their experience. Through surveying students, we looked at the impact on their knowledge gains, on their confidence in their ability to use that knowledge, and on motivation to complete assigned activities. And on the questions that we asked, to sort of tease out each of these thematic areas, we did receive very positive response. Now, these are, albeit, impressions of students. That’s sort of the first line of study available to us. And we feel pretty motivated ourselves by what we’re learning, by what we’re hearing from students. These surveys did also include some open text entry questions. And very interestingly, some of the criticisms that we heard were often this is a lot of time, this is a lot of work. But by the same token, we’ll hear in the very same response, but it’s very worth my time, and it really helps. And I feel confident when I go into the class, and I already know what my instructor is going to talk about, I feel more relaxed, I feel more able to pay attention. That kind of response has been very rewarding to receive from these student surveys.

John: With some other platforms, there have been studies that have looked at the impact on equity gaps, and they relate very much to what you just said, that basically, students come into our classes with very different backgrounds. Some come in with a very rich and strong background from prior courses, others come in with a somewhat weaker background. But to make it through an adaptive learning platform, the students who come in with less background can acquire mastery, but it takes a bit more time and effort. And the students who come in with a stronger background can race through a little bit more quickly. But much of the research does suggest that those equity gaps tend to be reduced. But it is at the cost of additional effort by those students who come in with a little bit less prior knowledge or prior experience with some of the material. And so I think that’s one of those real strengths of adaptive learning platforms compared to other formats like textbooks and lectures and so forth, where everyone is expected to move at the same pace. But as Chuck Dziuban, on a previous podcast has said, in a traditional course, everyone spends the same amount of time with the material, but they learn different amounts, when they’re working with adaptive learning platforms, they have to spend different amounts of time, but they all can achieve the same learning outcomes, they can all reach the same outcomes. But it may take more time to do so.

Linda: That’s certainly our hope, and some of what motivates the work that we’re doing. I feel like having respect for the time that our students need to spend is top of mind. We need to make sure that what we’re asking them to do is a good use of their time. And therefore this mode of delivery and the kind of content that we make available through the multiple activities and so on that Anna was describing. We’re giving our students opportunities to do that self testing that reiterative process and not simply trying to digest a narrative.

Anna: I always think back to a student I actually had in class a number of years ago, and this student went through the same class multiple times. And it was a class that was needed to get to other upper-level classes and was very frustrated, of course. And then we came to find out that this student in the multiple times of taking the class, the book was listed as required, but with either a digital or print textbook, you really don’t have any way to monitor that. You don’t know if the student’s using the book even if they have it. But this student had never got the book, just never occurred to them that, okay, this could be a really valuable tool. And so that’s really a benefit. I see. Like you were talking about equity, let’s give them this tool. But let’s make it something that’s valuable, that can really help them and if they need that added support, they’ll get it. But kind of level the playing field a little bit.

Rebecca: How do you see this platform evolving over time.

Linda: So we are able to work fairly closely with our platform partner, CogBooks. We’re able to contribute to their roadmap planning to some degree. We’ve collected feedback through a variety of channels, also, not only the student surveys that I’ve mentioned, but we also, in the past year, conducted a qualitative study that included student interviews and observations of students who were assigned to use PsycLearn. And this exercise gave us really valuable insight into how and where students focus their time and attention. And so for instance, through that process, we were able to get some real good insight into how they approach the summary of each module. And as a result, we’ll be able to give some greater attention, we’ve got some plans to enrich and make more accessible, the content that we placed there, because we were able to learn that they highly value that for later test preparation. So that’s just an example. We’re hoping to be able to continue to do those observational and interview studies.

Rebecca: So I’m really curious for each of you to answer this question. But what makes you really excited about PsycLearn?

Anna: I’m happy to go first, because I am a very energetic instructor. I will be at the front of the room, and I bring a lot of energy. And what makes me excited about PsycLearn is just the ability of students to be active consumers of their learning. I would love for them to take away something that is memorable, that feels unique. And so PsycLearn frees me up as an instructor, that I don’t have to go through every bit of background information in the way I use it in class, I can assume they’ve had things. Let’s spend time on things that are interesting. Let’s dive deeper into things. Let’s do more exercises for things that are challenging. And so the students get to think about and apply new ways of learning and engaging the material from the very first moment that they encounter the content.

Alan: Yeah, for me, it’s really about the inclusion of metacognition, I think it’s really helpful to have students be able to evaluate their own learning. I think this makes them do better on exams, and I think it makes them more successful in their entire college career. It’s also an important part of my own teaching. I have a lot of activities in class where students can answer and identify how well they know what they’re supposed to know. And what I often find is that students study, but they don’t know how much to study, they don’t know: “Do I know it as well as I need to know it.” And this product gives them a lot of feedback, really helps them understand if they understand the material as well as they need to. And PsycLearn really can do that in ways that regular textbooks cannot.

Linda: I would say that my excitement really comes from the big picture, the opportunity for us to deliver on these experiences that Anna and Alan are describing. But we’re going to be able to do that across the foundational courses in the curriculum. And so that larger picture future horizon is pretty exciting.

John: Well, you’ve already addressed some of this, at least in terms of PsycLearn, but we always end with the question: “What’s next?”

Linda: Well, we have sort of talked a little bit about future and what’s next and our opportunity to be iterative. The digital platform allows us to be continually rethinking what is the best activity to offer to reinforce or bolster certain concepts. And so that’s certainly something that is ongoing. We can take feedback from students on how well they feel various content is supporting them. We’ll be looking at what we can do to encourage a greater level and engagement with that support material, so that students aren’t feeling like, “Oh, that’s extra, I don’t really need to go there.” Indeed, it’s extra. But it’s really vital, especially for students who might sort of marginally understand. And so the opportunity for us to deliver a reframe of a concept can make all the difference at that tipping point.

Anna: Yeah, and I think what Linda had said about using the support material is really important to us. So we’re really interested in applying these learning science principles that Alan had talked about to say, “Okay, what is the best way to get students to engage with this.” We want them to see that cool stuff we’re putting on those pages, if they feel like they could benefit of it. So we want to reach the students that need that support. So I think just diving in a little deeper and figuring out the best ways to reach them, to get them to say, “Okay, yeah, I could use a little practice.” Or “I could use another example,” snd really assess that learning of their own. It’s a challenge. It’s a challenge we have constantly, especially when students are new at something, they tend to think, “Oh, yeah, I got this. I know it all.” And so getting better at that metacognition, I think, is really exciting to us.

Alan: Yeah, and I think for us, it has a lot to do with making a better and better product. I think about what it was like the first time I taught a course. And I equate that to the first iteration of a PsycLearn product. And then I think about what was it like the fifth time I taught that course or the 10th time I taught it, and how richer and deeper it was. And I think the same thing will happen with PsycLearn. We’re going to keep on creating a product that gets richer, more inclusive, and more likely to help students succeed.

John: I was really excited to hear about PsycLearn, and everything you’ve talked about makes me even more excited about the opportunity for this and I hope that other disciplines will start working on similar materials for their basic courses, because the benefits from this have been well established in terms of improving student learning and reducing some of the equity gaps. Thank you.

Linda: Thank you.

Alan: Thank you.

Anna: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

205. Tutoring

Equity gaps in educational outcomes play a major role in perpetuating economic inequality. In this episode, Philip Oreopoulis  joins us to discuss his research examining how tutoring and computer-aided instruction can be used to reduce disparities in educational outcomes. Philip is a Distinguished Professor of Economics and Public Policy at the University of Toronto, the Education co-chair of MIT’s Abdul Latif Jameel Poverty Action Lab, and an award-winning researcher who has conducted a wide variety of studies relating to education and educational policy.

Shownotes

Transcript

John: Equity gaps in educational outcomes play a major role in perpetuating economic inequality. In this episode, we discuss research examining how tutoring and computer-aided instruction can be used to reduce disparities in educational outcomes.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Philip Oreopoulis. Philip is a Distinguished Professor of Economics and Public Policy at the University of Toronto, the Education co-chair of MIT’s Abdul Latif Jameel Poverty Action Lab, and an award-winning researcher who has conducted a wide variety of studies relating to education and educational policy. Welcome, Philip.

Philip: Thanks so much for having me.

Rebecca: Today’s teas are:…Philip, are you drinking tea?

Philip: My tea is coffee. I love coffee. I once looked for a reason not to drink coffee, I couldn’t find one. I love my black coffee.

Rebecca: A true researcher at heart. [LAUGHTER]

John: And I am drinking a bing cherry black tea, a custom Tea Republic tea made for Harry & David.

Rebecca: And I have Irish breakfast tea. I really need to get some new tea [LAUGHTER]. I’m going to a tea store this weekend, so I’m looking forward to getting some new options.

John: And we have lots of tea in the office, some of which may not be as fresh as it was a year and a half ago. But this one still is good. It was purchased right before the shutdown.

Philip: You guys are inspiring me. I think I’m gonna have some tea sometime today.

Rebecca: All right, good, good.

John: In a November 2020 Scientific American article, you describe a meta analysis that you worked on with some colleagues that found that tutoring results in significant improvements in student learning. Could you describe this meta analysis a bit and what you found?

Philip: To backtrack a little bit, how it got started: my colleagues at J-PAL, Vincent Quan, Andre Nickow, and I, had heard about the potential of tutoring to be an effective form for increasing test score learning performance. For example, there’s Benjamin Bloom’s seminal article in the 80s, where he had two very small studies done by his students that both found off the charts improvement from offering tutoring in randomized control trials. In fact, that’s why he called it the ‘“2 sigma problem” that he found estimated impact from these two small studies were raising learning performances by enough to potentially solve most of our problems that we would be having in education policy. There were a number of recent studies as well, a randomized control trial coming out from the University of Chicago’s Ed lab, also finding very promising results from an RCT looking at providing in-class tutoring to grade nine students. And so we wanted to explore whether there was some consistency in these results, so we decided to try to take a more systematic look, and we gathered up all the RCTs, randomized control trials, in the last 40 years for about 96 studies, and we took a look and we found that consensus was quite remarkable. About 80% of those studies found significant effects larger than .2 of a standard deviation, and the average effect size was .38 of a standard deviation, which is like the equivalent of almost an entire extra year of school, from receiving these programs. And not only were the impacts really quite meaningful, as about as large as you get from education interventions, but they were consistent across the board. I think that this is about as much consistency as you’re ever going to get in an education policy intervention. So we were quite excited about that. We found that the effects were pretty consistent no matter which type of program that you looked at. They were larger for things like in-school delivery, three days a week, one-to-one delivery, full time tutors, but even in cases where that wasn’t the case, usually there were still significant effects.

Rebecca: Can you talk about what age the students were, what grades they were in?

Philip: It was for K-12.

John: I think it’s probably safe to assume, though, that the same effect would hold in the college environment as well. Those are some pretty dramatic effects.

Philip: Of course, to some extent, maybe it’s not that surprising. Giving instruction one-to-one leads to higher learning gains, and the biggest challenge, of course, is cost. We can’t all have our own teacher when we go to school. And so the biggest challenge, which gets back to Bloom’s point calling this the 2 sigma problem, is I think we have a powerful intervention to help education, it’s just that it costs too much to implement it on a larger scale. So the fundamental problem is to figure out a way to scale this in a way that can complement the classroom instruction.

John: And so that’s one of the things I think you’re looking at now, how this can be scaled up in a more cost effective manner. Could you tell us a little bit about your current research in terms of computer-assisted learning?

Philip: Sure. So computer-assisted learning or computer-assisted instruction is a type of educational software designed to help students progress through topics at their own pace. It has a lot of similar features as what you might receive when you’re receiving tutoring. So a typical example might be Khan Academy, MATHia, there’s lots of other types of software designed to help with different topics, math and reading, but they all have these sort of common features that allow students to progress through topics at their own pace. You receive immediate feedback from trying to work through your own problems and a chance to understand where you went wrong. If you do make a mistake, there’s data that’s generated from going through it that someone like a teacher might be able to follow and respond to. And so computer-assisted learning can, in some ways, simulate the tutoring experience, but of course, at a much lower cost. The challenge is you don’t have a real person guiding you through it. So even though a platform like Khan Academy is easily accessible, your willingness or motivation to go through it on your own is probably not as great as if you had a real person guiding you through the same material. So there has been some experimental evidence on computer-assisted learning, not as much as theories on tutoring, but of the 15 or 20 randomized control trials that have been done in this area, they have also been showing quite promising results. In cases where computer-assisted learning is provided, especially during a school setting, those receiving it also seemed to be performing at significantly higher rates than those in the comparison group. So there does seem to be some promise at using computer-assisted learning to generate the gains that we see from tutoring. But the way to introduce it, the instructions that teachers need to learn how to use it effectively, are not yet maybe as developed as we’d like them to be. So getting to, I guess jump into what I’m working on, I think that there’s a lot of potential for leveraging existing resources to combine with computer-assisted learning in a way that might come close to the tutoring experience. And so what I’m thinking of is in the classroom, that the kind of facilitated practice that might go on, say, in a math subject might be much better through a tool like Khan Academy than paper and pencil that we often give students. And so the question I’m investigating is around reshuffling the classroom in a way where the teacher is trained how to use computer-assisted learning more effectively in the classroom to generate that type of experience. So in the context of the program that I’m looking at now, which tries to integrate Khan Academy more into math classes, the teacher is still instructing and presenting topics, but now emphasizing the students following an individualized roadmap that allows the students to progress at their own pace, rather than having to keep up even if they’re missing on topics and not understanding. So the program which we’re calling “Coaching with Khan Academy,” or CWK, has students receive a roadmap of incremental topics and videos to follow at the start of school that roughly proceed in the same order that the teacher is going through. Now, the teacher has the students to try to work on this roadmap for at least an hour, an hour and a half a week, and tries to facilitate that time during the class and encourage more done at home, and the students then have the ability to hopefully get into a routine of watching a video and taking the exercises, and if they don’t score high enough on the exercises they’re asked to try to understand why they made the mistake using the hints and tips and guidance that Khan provides or gets help from the teacher, and then repeat it so that they don’t move on to the next topic until they’ve mastered that. So the students are not proceeding all in the same pace, but it is just a much better way to learn math such that the students don’t go on to the next topic until they’ve established a strong enough foundation on the first one.

John: During the global pandemic, most high schools moved to emergency remote instruction for an extended period, and there’s quite a bit of evidence that that led to a decline in overall learning, but also some growing achievement gaps which are tied to household wealth and the wealth of the school districts in which the students reside. What types of policies could be implemented at the K-12 level so that students are more equally prepared for entry into college

Philip: On COVID, we’ve all been exposed to online learning now, and most research suggests that it’s not a great substitute for in-person but there are certain benefits from being able to speak with a real person over a computer in regards to tutoring. So the biggest one is convenience, both for the tutor and the tutee. It’s nice to be able to jump in on a call and spend just 30 minutes on that or an hour, and not have to drive to the person’s location or do this after school. The opportunity to facilitate more tutoring, I think, is increased by having this online access. So I think there’s a lot of interesting promises from that. This one particularly interesting study that was done during COVID last summer, where a group of Italian faculty organized a volunteer tutoring experiment where they got the Deans of their respective universities to invite university students to volunteer their time, three to six hours a week to reach out and connect with students who have been struggling in the high schools and lower grades. And on the flip side, they got the school districts of several locations in Italy to ask teachers to identify students that they thought could benefit from having this one-on-one instruction. And then the response was great in both ways, there were a lot of people willing to volunteer their time for this effort, and there was also a lot of perceived need for students that needed this. And so from this large set-up, they randomized who they were able to give this offer of assistance to. And it was done all online, sometimes over the phone, but more often through Zoom, or Skype, or whatever was most convenient for the match to take place. The tutors met with tutees, for three hours a week, over six weeks. The topics were either math, Italian, or English, and then at the end, the researchers collected the survey and found similar gains to what we were finding in the online overall. Not only that, but they also collected data on mental health and found improvements in feelings of connection, more positive outlook on life. And what’s also interesting as they seem to show improvements and positive outcomes for the tutors themselves, as well. So it stands the potential for a win-win, and this was all done online. So it’s like the only online study I know, but it seems to show the potential that it might be done there. One other example I should mention is Khan Academy has also initiated another organization that facilitates free volunteer online tutoring. It’s called ‘schoolhouse.world’ and it’s been interesting to watch that trying to get up and running. Their system allows anyone in the world to volunteer their time as a tutor, and then they try to connect anyone in the world wanting to receive that tutoring. And you get some sense of some of the challenges from doing that. How do you screen for quality? And also, how do you screen for safety? So they’ve had to go away from a one to one model to more of a group model. They’ve had to have systems in place to check the quality of the tutoring, what’s being discussed. They’ve had to switch to allowing only high school students to receive the tutoring and a few other challenges. And so there’s challenges but also a lot of potential in this that wasn’t available from always having to meet your tutor in school or after school or face-to-face. So the potential scalability is enormous, and that’s where the intriguing possibilities are with that tool.

Rebecca: So if we’re looking to reduce achievement gaps, we’ve talked a little bit about COVID and the mix of instruction that students might’ve had during COVID, the quality of instruction, access to technology, to even have interactions with teachers in some cases, and historically even, differences in ability when students arrive in higher ed. What are some of the things that the higher ed community might be thinking about in terms of this research? Should we be advocating for certain kinds of policies or programs in K-12? Should we be trying to institute some of these things in higher ed? What are your thoughts on that?

Philip: So just in terms of advocacy and thinking about facilitating more equality, there’s no question that tutoring has, in general, been an unequal program. There’s the whole private sector of tutoring where a lot of households for more affluent families seem to receive it than those from less affluent households. And so one thing we can do as policy-makers is to try to facilitate more tutoring to happen in schools, especially at schools for more disadvantaged backgrounds. We can also focus on providing tutoring to those who need it most. I think that there is a growing awareness of the potential for tutoring to make a real difference in helping address the learning loss that may have occurred with the pandemic and just helping address education inequalities in general. And so a lot of resources have started going towards trying to increase the amount of tutoring happening in schools. I think that the more we understand how to implement it successfully, the more guidance that we can provide the K-12 sector in trying to introduce that. I think that there is a lot of optimism now around its potential. I think tutoring is one of the most effective programs that we can offer to make a meaningful difference at scale, such that we can get more students arriving into post-secondary ready to handle it and succeed well there. So that’s on that end. I think that there’s no reason why we also can’t consider tutoring at the post-secondary level as well, and the potential benefits that might come from that. Even if we just look at first-year calculus, or other subjects in math, computer-assisted learning is well developed even at that level, the need for tutoring at that level is there as well. And so it really does go from that importance of establishing a foundation that one might benefit from tutoring at earlier ages. But even at the post-secondary level, regardless of what level the student is, we can all benefit from one-on-one instruction compared to being in a calculus class of 500, right? I think there has been less research that’s been done in that area, but the evidence certainly points to the direction that tutoring at the post-secondary level would be also effective and important to consider.

John: And you mentioned that Italian experiment where college students were providing tutoring, and you mentioned that that was a very positive experience for the college students as well. That might be an interesting model where college students could improve their own skills and develop a bit more automaticity and more practice in basic concepts, while helping bring students up to a higher level in secondary schools. That’s a program that I think offers a lot of potential.

Philip: So I would agree, absolutely, the expression is you don’t really understand something until you teach it. I think that there’s something to be said for that. I think that there’s also a lot of skills and experience that is gained from trying to help others, from trying to connect with perhaps younger individuals that have not had the same background as you. I think that the experience is also attractive to employers looking at who to hire. I think there’s huge gains from all the things that you might volunteer or use your time for in college, spending some time to volunteer to do something like tutoring could be a very rewarding thing as well. So I’m also excited about that model. I think that there are ways to try to facilitate that kind of model at scale and more research needs to be done to explore how to do that.

Rebecca: One of the things that I heard you mentioned early on in the conversation is the idea that, historically, folks who had access to tutoring are more affluent. So the students who most need the tutoring are the ones that aren’t always getting it, because they can’t afford it. So I love the idea of having it in schools or it’s a part of our programs. But also I think sometimes tutoring has a negative connotation to it. It’s like a deficit model. Especially I’ve seen this in higher ed, students don’t want to go to a tutor because it makes them feel like they’re dumb or something.

Philip: My first reaction to that is that tutoring can be beneficial at any level. For example, in the Khoaching with Khan project that I’m looking at, the potential is to help all students in the class regardless of their level, because every student can be given their own individual roadmap. And that not only includes those that are behind grade level that benefit from establishing a stronger foundation in that earlier material so that they can catch up, it also includes those at a higher level that don’t have to be held back or wait for the instructor to cover new material can use a platform like Khan Academy or a tutor to work on more challenging material that interest them. And so how to remove that stigma that exists in general, I agree the usual perception is when someone asks, “Do you need a tutor?” it’s because you’re struggling. It doesn’t need to be that way, but at the same time, I think the more we become aware of the benefits from the tutoring, the more we realize that it’s a great resource to take advantage of. Getting back at the college level, I don’t know about your own experiences, but it always amazes me how few students take advantage of all the free tutoring that’s being offered by the universities through, like, office hours. The opportunity for receiving one-on-one discussion is often there, and yet so few students seem to take advantage of it, perhaps because of that stigma or perhaps they’re too busy. Some of us, when we went through college, were pleasantly surprised by how much you can get with office hours of graduate students and extra tutoring and how much you can learn from that process.

John: As in a lot of classes, students are treated as if one-size-fits-all education and students come in, especially in subjects such as math where there is a very rigid structure, if you don’t have a solid foundation and concepts, learning new topics is not going to be very productive, because you don’t have that foundation to connect to. And I see that in my own classes, and it’s a bit of a challenge to try to do that. Because of issues of scale I often teach large classes, I try to rely on peer instruction as much as possible with small group activities. Could small group peer interactions in working through problems and problem sets achieve something similar to the one on one attention?

Philip: In the literature, it’s called peer-to-peer, we did not look at peer-to-peer in our meta analysis on tutoring, but there is some literature and there’s some effort to consider that. It’s a little bit of a different model, because you’re relying on slightly older students or similar students to help assist other students. I think more research needs to be done on how to make that happen effectively. On one hand, the potential is there to make this a scalable, effective program that doesn’t cost very much. On the other hand, monitoring quality and the potential to train to be a tutor and to do a good job with it may not be there as much as with the regular type of tutoring program.

John: In particular, I was thinking of activities in class where students work on problems in groups, and they try to argue out solutions. They work together and they can explain to each other things they don’t understand, but the key aspect of that is they get feedback on whether they’re correct or not, some constructive feedback on where they went astray. But I was just thinking that those types of small group interactions could provide some of the benefits without that stigma of needing to go to tutoring and perhaps at a higher scale than tutoring might work.

Philip: The advice that I often give my students is to study until you feel you can explain it to someone else. And so there’s a similar, perhaps, mechanism at play when we’re thinking about that. When you try to write down a concept or explain it, even to yourself, out loud or to someone else, you quickly realize what you understand and what you don’t. There does seem to be a lot of potential there.

Rebecca: Sounds like one of the keys to reducing stigma around all of this is making the coaching or this tutoring model just something that’s normalized. Maybe it’s normalized in class, it’s normalized through the school day, and then people might be more apt to take advantage of it because they have access to it. But also, it becomes a standard way of being, that’s what other people around them are also doing.

Philip: Absolutely! I think if we can reframe tutoring as just individualized instruction or personalized instruction, then we can all understand the potential benefits of receiving more personal help than in a classroom setting, and that goes for pretty much anyone.

Rebecca: It really also matches up well with a lot of universal design for learning principles of flexibility as well, and allowing students to go at their own pace and finding ways of teaching and learning that match well for students and where they’re at.

Philip: And of course, the issue is scale. Getting children to learn in a classroom of 25 to 30 students, when these students vary enormously in academic levels, is just really difficult. And trying to figure out a way to provide that individual attention is the challenge that all teachers face and have been facing for many, many years. And if we can find a way to scale adding on or providing more and more individualized attention, it has the potential, I think, to make a real difference in education. Of all the potential policies that we can be looking at, I do think that, at the school level, leaning towards more individualized instruction is where we should be looking at, for a solution.

Rebecca: It’s so interesting to me that we’re having this conversation early on in our semester, because after teaching online for a year, which I hadn’t done previously, I’ve really worked to make my classes more flexible and actually offer some of those kinds of models that you’re describing where students are going more at their own pace, and that they can get some individualized instruction when they need it and that they need to do this mastery learning so that they build on things over time. It looks to me like maybe I need to look more into tutoring and coaching models that have worked really well to see if I can’t implement some of that more during class time.

Philip: There may be different ways to do it. Some may be more effective than others, but I do think, getting back at what John was saying, it’s harder to provide that individual support or help to students arriving in college without that foundation. I have done some other work at the college level, trying to facilitate more personal attention to students arriving, trying to help them out and encourage them to get into better habits, and it has proved quite difficult to change behavior, and so I have found myself reacting to that by focusing more on earlier grades to see if there might be more promise on trying to foster better study habits, better learning habits, earlier on with the hope that students arrive in college more prepared.

John: I think that’s one of the things a lot of behavioral economic studies have found. Interventions that result in long-term changes of behavior are challenging in general.

Philip: Absolutely.

John: And I think you’ve done some research on that.

Philip: Absolutely. So if we have to change one-time actions, like helping students through applying for college, applying for financial aid, those types of interventions are much more promising at affecting one-time goals than to change habits or routines that involve much more continuous behavior. So helping someone study more effectively, spend more time studying, these are much harder problems to solve. And maybe low-cost nudges that we’ve been looking at in the literature may not be as effective. I think that does tie back into how my perspective has changed over time. It’s hard to have significant influence without personal connection. It’s a lot more expensive, but there’s only so far you can go with sending an email or a text message or a one-time meeting in trying to change someone’s learning trajectory or life trajectory. And the more you sort of look at education policies that have been successful, the more you notice that they often come with this personal connection that’s been important for making that meaningful change.

Rebecca: It seems like we should all be really advocating then for these much more early interventions. It’s much more cost effective if we get those habits in place really early [LAUGHTER].

Philip: I will say there’s surprisingly not enough research on the long-term effects of tutoring. I’ve seen one study that has found that the benefits of receiving that tutoring continued one year past the program ended; the effects faded, but not by that much, and that’s the only study I’m aware of that actually does a long-term study. So on the question of whether we can have these life-changing impacts from targeting earlier ages, certainly, there’s a literature for the very young… like, almost helping at the household, but at the school, I think that more work could be done.

John: And that could be a really productive research area. Before we started recording, we were talking a little bit about, with the pandemic, creating our own videos. Could you talk a little bit about how you try to implement what you’ve learned in your own classes at the college level?

Philip: Yeah, I think that using the situation last year to put my lectures online has freed up space in the actual lectures to be more interactive. So I think it was a benefit both ways. The videos of the lectures themselves became more streamlined, I got a chance to break them up into smaller parts, sort of like Khan Academy videos, where instead of one video that’s two hours long, that goes all over the place, and you’re staring at me and the Blackboard, I created five- to ten-minute videos of vignettes that I could focus on with slides and have a series of these videos that students could watch at their own pace. I could edit them and make sure that the video is as succinct as possible and gets across what I really want to say. So that was good on the video side, and then on the actual lecture side, we spent that time going through problem sets and answering questions and it was much more interactive, closer to the spirit of more personalized instruction. So there was more opportunity for questions, more opportunities for the students to get more involved, and I think it did lead to more satisfaction of that approach. Obviously, the big question is, ‘Do they really watch the videos when they’re asked to do it on their own?’ I think there are ways to try to incentivize that, but just like any class, the students really perk up when they’re working on a problem that was, say, a previous exam question.

John: I’ve used a very similar approach. I’ve used videos for like 20 some years in my classes, but one thing I started doing last year is I embedded questions in the middle of the videos, and that’s a pretty effective incentive structure. It does get them all watching the videos, and at least thinking about it and trying to make some connections while they do it, and that’s worked pretty well.

Philip: Not only that, but you can make them mandatory for class participation. So you stick those questions in and they have to watch the video to find the questions when they pop up, there’s software that can do that. And then you can make it as a way to encourage them to have to watch the video.

John: Do you think that more use of computer-aided instruction is going to be helpful in allowing more students to be successful?

Philip: I’m very optimistic on this potential of leveraging computers with teachers and parents working together on trying to facilitate high-dosage practice. We’ve been talking mostly in math, but it could also be language as well, and maybe other topics. But I think this really is a good way to learn, as long as the practice time is long enough, and the student’s not stuck. I think that it takes a while to get into the habit, getting used to the software, getting used to the routine, both for the teacher providing this and for the student doing it, and so that, for me, right now, is the biggest challenge. I am optimistic that if we can facilitate a way to help teachers and students get to that higher-dose practice using computers, then very good things will happen. I think that the evidence is highly suggestive that the high dosage is a worthwhile thing to get done. I’m hoping that we can generate evidence that that’s the case, but we are finding that there are challenges because there’s a learning curve, it is changing the way that the classroom is done and changing the way the student usually learns, but I’m optimistic that if we can get past that, the students and the teachers will come to like this approach, and that we can do more of it at scale.

John: And I think a lot of people began experimenting with some sort of a flipped approach where they created videos and then use the classroom for more interactive activities, ast least at the college level, I don’t think that’s happened quite as much at the secondary school level. But I think that has helped provide at least some professional development for faculty. But it is an adjustment that students are not adjusting to perhaps as easily as I would like, I know I always have trouble getting across to students that there is some benefit of working through problems in class and watching videos and learning some of the basic concepts outside of class. Students would rather be lectured to, there was that big study that was done at Harvard not too long ago, where students were asked about active learning classes versus lecture classes, and the research certainly showed that active learning in the classroom led to significant learning gains, but students perceived a higher learning gain from lecture classes, and that’s where I think that issue of students’ adjustment is a challenge, and until we get to see a large amount of this occurring, it’s going to be a while convincing students of this, because it’s really easy to sit there in a lecture and nod and smile and have it all make sense and it seems to fit together very logically, but then when you try to apply it, there’s a bit of a problem, and then the questions are somehow unfair. But when students are faced with problems and interactive work in class, they’re confronted by not knowing things as well as perhaps they thought they did, and it’s not as pleasant of an experience. And I think that’s the source of that metacognition, that students perceive that lectures are more effective, because it’s easy to sit there and listen in, and it all seems reasonable. But the problem is when they try to work through problems and realize they don’t quite have those connections fully there yet.

Philip: The lecture seems to make so much sense until you sit down when you get home and try to go over it again, but I do think there’s the potential for this middle ground that even in the experiment we’re looking at, we’re not entirely flipping the class, in fact, we want to work with the teacher to understand what their own preferences are, while still trying to hit this high dosage of practice, which may occur in class, but also could occur at home as well. And I think that there is something to be said by having a lecture of a new topic being done in class, in person, with the real person. It gets back to that importance of personal connection that the computer is not able to provide. And so maybe there is a sweet spot around providing real instruction, real empathy, but also enough time to be working through these problems at your own pace. My vision for the Khan project is that students say, in grade four, getting 90 minutes of math a day, maybe half an hour of that would be the teacher’s own instruction of a new topic, but then a lot of the other time would be students working on their own devices, while the teacher takes the time… instead of just sitting up at their desk… walks around and spends a lot of time looking over the student’s shoulder, using the data that they’re seeing to understand who’s struggling and where, and spends a lot of time working individually while the student is using the computer. So there’s still that interaction going on and taking advantage of the personalization. I think they too can go really well together.

Rebecca: That’s definitely something I’ve been experimenting with. I went all the way flipped before, and right now I think I’m right in the middle. There’s some flipped, there’s some demos that are live so that people can interact and ask questions, and then there’s lots of practice with individualized attention. And it does take a little time to get everyone on board, to get everyone trained to do things in a new way. So in a 15-week semester, it might take two full weeks to develop new habits and workflows for everyone, but really after we get over that two- week hurdle at the beginning of the semester, my classes tend to settle into a routine that seems really productive and that students have been pretty positive about.

Philip: A key feature of the coaching with Khan program, is that every teacher gets their own coach that we spell with a “kh,” and our coaches meet with the teacher prior to school to go over our suggested recipe to follow, but then they don’t just leave it at that, they keep working with the teachers to check in and try to troubleshoot or brainstorm or reassure and remind the teacher until things are going smoothly. But it can take longer than two weeks to figure out how things are going, and then on the student side, it can take a while for them to adapt and understand that there’s some independence on their own for wanting to do it. The hope is that the students start to gain confidence when they see their own progress, when they see that maybe they didn’t consider themselves a strong math student, but if you start them at the right spot on this roadmap, and then they proceed incrementally, and they can see that they are advancing, then they start to understand the potential benefits and internalize the desire to keep going on their own.

Rebecca: Yeah, that autonomy and that empowerment, I think, is really key to the whole puzzle. And I think something that probably tutoring historically helps students achieve is that they can do this. They might have a little extra guidance initially, but then they achieve it and can do it, and that’s really empowering.

John: That’s our hope

Rebecca: We always wrap up by asking: “What’s next?”

Philip: What’s next? I think I made some notes on that. [LAUGHTER] So I think the issue around tutoring and individualized learning is all about, now, scale. I don’t think we need another study to demonstrate that one-on-one instruction, or one-on-two is an effective additional tool for learning, that more should be done if it were possible. A lot of resources are now going into trying to provide individualized instruction. I think a lot of policymakers and governments are looking to tutoring as a way to address some of the learning loss that may have gone on during the pandemic, and I think, in that space, there’s some optimism by researchers and policymakers to try to understand what types of scale up are better than others in a way that we can make a meaningful difference at the aggregate level.

Rebecca: Well, thanks so much. I’m really excited to hear more as your research develops and more information becomes available!

Philip: It was a pleasure to get a chance to chat with you guys. It’s a topic I’ve been spending a lot of time on and losing a bit of sleep on trying to get things to work. The experiment that we have going on, this is going on in Texas, and one of the challenges of doing a field experiment is that so many things go wrong while you’re trying to deal with real people, real students, and provide evidence that this is a good idea. And it’s always a bit frustrating to face these challenges, like just account issues, students have trouble getting on to Khan Academy and the teachers getting frustrated, and it would be a shame to have those issues that can be worked out actually create this wedge from the program going smoothly and making the difference between having these great impacts or not. So it is stressful, but I think it’s worth it to try to keep at it, and I hope to be able to do so. With funding and policy support we’ll just keep trying. I think there’s a lot of interest in it, I think that it hasn’t been difficult to motivate these ideas and wanting to do more on it. So thanks a lot for giving me the chance to share these thoughts.

John: Your work is incredibly important. And so much income inequality is associated with differences in educational attainment, that understanding these achievement gaps and what we can do to narrow them can have a really dramatic impact on society.

Philip: Fingers crossed!

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Anna Croyle.

[MUSIC]

48. The Culture of EdTech

As faculty, we engage with education technology as it relates to our classes but rarely consider the larger EdTech ecosystem. Dr. Rolin Moe,  the director of Academic Innovation and an Assistant Professor at Seattle Pacific University, joins us to discuss the politics, economics, and culture of EdTech.

Show Notes

Transcript

John: As faculty, we engage with education technology as it relates to our classes, but rarely consider the larger EdTech ecosystem. In this episode we examine the politics, economics and culture of EdTech.

[Music]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[Music]

Rebecca: Our guest today is Dr. Rolin Moe, the Director of Academic Innovation and an Assistant Professor at Seattle Pacific University. Welcome, Rolin.

Rolin: Thanks for having me.

John: We’re glad to have you here. Our teas today are…Rolin, are you drinking tea?

Rolin: I am John.

Rebecca: Yes!

Rolin: I am having the Maui Up Country blend that I picked up on a on a vacation that I had brought for the office, and we ran out. So I am drinking the wonderful Keurig inspired Celestial green tea today. But I am joining you guys over there. [LAUGHTER] What are you guys having?

Rebecca: I think it’s a green tea day. I’m having black raspberry green tea.

John: …and I have a ginger peach green tea.

Rolin: Excellent.

Rebecca: We’re all in sync without planning, so that’s nice.

John: We invited you here to talk a little bit about your April 2017 EDUCAUSE Review article (which has created a little bit of a stir) where you were talking about the growth of educational technology in higher ed. What types of EdTech in particular were you talking about?

Rolin: So, John that’s a good question… and a little bit of preface on the article itself. I wrote that with George Veletsianos, who is Canada Research Chair in Innovation and a Professor of Education and Innovation at Royal Roads University in British Columbia. We started this project in 2013 at a time when MOOCs had just come into conceptualization. Laura Pappano noted that the year before had been the “Year of the MOOC.” John Hennessy at Stanford said that the MOOCs were going to be a “tsunami that was going to wash away higher education as we knew it.” Clay Shirky compared higher education to a rotting tree that was in need of a lightning strike and this was going to change it… and so, this very optimistic (to the point of Pollyanna) thought on educational technology. And George and I both, as people who are scholars and practitioners in educational technology, were a little taken aback by this. The promises that were being related to educational technology didn’t match the literature. The history of educational technology didn’t match the present and the future track of these innovations, based on their previous experiences (kind of Silicon Valley startups) was not a positive one. As I mentioned, we started writing this in 2013 and the landscape kept changing. Ownership would change, or business models would pivot, and we had to rethink what we were doing. So we kind of, instead, came back to this more systematic review of what is educational technology, or EdTech, and we thought of it in socio-cultural terms as a phenomenon. So, thinking about that, it’s not necessarily a product that we are providing critique for but it’s more of the idea that by bringing products in, whether they be cloud based softwares, learning management systems, apps, learning technologies interoperability, or LTI, or outsourcing it to a third-party vendor, whatever that vendor may be. That approach cannot be thought of as altruistic in and of itself, but it is built in society that is usually, at best, tangential to education, but often completely separate… being brought in for profit bearing reasons, whereas our institutions, by and large, are education-bearing institutions that are looking to gain enough profit to continue operation. So, what EdTech are we looking at? We really want to be creating a more critical consumer of all EdTech. And you can definitely see that today in privacy issues that are coming out with Facebook and algorithmic issues that are happening with Twitter, and discussions of what constitutes free speech or hate speech on these platforms. When we wrote, we were much more thinking about the technology that’s getting into schools, but even there, some of the things that are happening in K-12: the data from these students is not necessarily protected, whether it’s getting hacked and sold to other places or if the companies themselves have connections to other products and other vendors. So, it’s a really meta piece to be thinking about. I don’t necessarily have an axe to grind with any particular software. That’s why we were very software agnostic when we were writing the piece. We just really want to be much more conscious of how we’re using technology in our teaching practice and what is happening because of the technology we’re bringing into our classrooms.

Rebecca: Thanks for laying down that groundwork. I think that foundation is gonna give us a good ground for discussion today and will help our listeners know exactly where we’re starting.

John: A lot of these things, where people were really optimistic about the introduction of MOOCs and so forth, we’ve seen all this before. Television was going to do the same thing. Before that radio was, if we go back further, printed books were going to have this big impact. So, these are issues that have been around for a long time. But, you focus on several issues that, perhaps, are more pressing now. One of the things you talked a little bit about is how colleges have been pressured by economic circumstances, by rising tuition costs and pressure to keep costs lower, to rely more on these external vendors. Could you talk a little bit more about that aspect?

Rolin: Absolutely. I need to preface here again, John, I appreciate you bringing up television. Because there’s a time that a lot of institutions invested in broadcast studios, with the idea being that we were going to be able to amplify education and we’re going to be able to have closed-circuit educational opportunities at senior centers and satellite campuses. And so you have in many land-grant colleges these forgotten studios, that in some cases are now being turned into teaching and learning centers where you have a green screen and you can show what you’re doing in Canvas, or Desire to Learn, or Blackboard, or whatever the system is that you may use… Moodle, I don’t want to leave anybody out. But, to think about my experience as an educator, I have a connection to this particular podcast. I cut my teeth as an educator in my first career, which was in film, at Duke University’s Talent Identification Program where I got to know John Kane, who has been kind of very foundational in how I think about teaching and learning. So John, thank you for that, and it’s wonderful to be on your show. We’ve seen all of this before and we failed to learn our lessons in education. So, we didn’t get out of television what we thought… what we thought we’d get out of radio we didn’t get. It’s important to look back and see “Well, what didn’t happen that we expected to happen? What did we plan for? What was the consequence? What were the unintended benefits? and what were the unintended pitfalls?” The problem, or the big difference today, is a lot of the technology is being looked at from an efficiency standpoint. So, television and radio and even if you go back, like you mentioned with the printed books, you go back to correspondence courses and using the Penny Post in order to be able to give keyboarding instruction for secretarial jobs. So, those technologies were based on much more inclusivity in education. You had a technology that made education available for more, and you had an opportunity to get away from geographic distance as what was keeping people from school. With digital technology what we’re seeing now is almost an inverse relationship that “Yes, we have this opportunity and we sell it.” So, the MOOCs were sold as an opportunity to democratize education for everybody. But, this is really framed in a cost-cutting perspective. That we’re going to bring in technology to keep costs down. That’s very important, costs in education, and higher education especially have skyrocketed, and to think about how we can be looking at this. But it’s disingenuous to say that our digital technologies are going to democratize education for all when we want to use them to save money more so than grant access. We have to look at both critically. We have to put the same research behind both. Moreover, what’s happening when our use of technology is in the gaining of data analytics that could be used, at best, in our spaces, but at worst by third-party vendors that we’ve signed contracts with that we don’t truly understand where they’re going or where they’re taking these things. So, I started with your question and went in a lot of different directions I’m realizing. But, I think it’s important to do that historical review and think about all those places because there’s a desire there, with what education’s supposed to be, if you want to think about Enlightenment-based thinking on education. But, we are at a different point now than we were with what someone like Soren Nipper would have called generations of technology. The first generation being radio, the second television, the third digital. This fourth kind, of web 2.0, has a much greater economic impact, both on the institutions as well as the whole purpose of education. That’s something that we don’t see a lot of in the literature and something that compelled George and I to write this article.

Rebecca: I’m hearing you talk about the the desire for more access but then also these rising costs. If we’re using EdTech, are students actually just getting more access? or are we just making things more expensive at the cost of actual learning?

Rolin: Yes. [LAUGHTER] It’s difficult because in some cases there is an upcharge on taking the course online. And there’s good reason for that because in order to teach a course online, if I’m an administrator, I now have to think about a faculty member who’s going to be working through that course. I have to think about any licensing that I need for contents. So, making sure that my reserves in the library can be easily flown into my LMS and that I have the rights for reproduction in that space. I have to think about instructional design, I have to think about information technology. I have this much larger infrastructure that’s involved, depending on what I’m doing: if I’m going to be using an anti-plagiarism software; if I’m going to be using an online proctoring software, a special grader, a video library of contents. There are four or five different buckets of LTI and those are the general ones, not anything discipline-specific. So, that brings this cost up. At the same time, if you think about Moore’s law, and as technology is increasing and the capacity to do things continues to increase, traditionally we have seen costs go down in this model. That hasn’t happened with education. So you have a space where students are presented in media and, I would say in a lot of cases by schools themselves, that this online efficiency opportunity to engage is going to bring your cost down, but then your cost is becoming more, because the cost on the institution is more. All of that is to say, at some point, if you’re gonna be selling both cost savings and access, that’s not a recipe for success. In many cases, we have the access, but it’s not to people, it’s not to a high impact educational experience that you have come to think with a stereotypical higher education space. I think of the Sally Struthers ITT Tech, you know, where you can do the courses in your pajamas. So we’re giving access in real time to curriculum and to materials, are we necessarily giving it to really engaging learning activities? In some cases, yes… but I don’t think the literature would say that those brightest cases of access are meeting that romanticized version of what it means to be a student in higher education. In many cases the most successful institutions in creating access and bringing costs down are the ones where faculty have been replaced by kind of quasi-administrators who work as admissions support specialists, tutors, retention specialists, program developers, and fundraisers. Kind of doing all of that from an office space, and that looks remarkably different from what we see in cinema, as somebody who works in film studies… what we see in cinema as that college experience. So, we’re gonna have to rectify what it is we think college is supposed to be with what it is we’re selling it as.

John: Might some of that be that, with new technologies… giving an example from economics… when steam engines were first introduced, we didn’t see any real improvements in productivity for decades after that. When the internet was first introduced and people shifted businesses to that, it’s taken decades before we’ve seen much of an increase in productivity. Is part of it that we try to use the new tools in the same way that we traditionally taught and we haven’t learned how to use it more efficiently, or is it something inherent in the shift to more digital media that limits the interaction between the instructor and student and may limit learning somehow.

Rolin: John, thank you for bringing that point up. If you think about professional development technology, the stereotypical overhead projector that is used to present material is then replaced by the PowerPoint…and what was interesting is, in some cases, the first uses of PowerPoint in classrooms (because of bandwidth issues) were printouts of PowerPoint slides that were then put onto overhead transparency. So what we see in many cases today what constitutes online learning is the lecture based approach the “sage-on-the-stage” model of teaching where we’re using our learning management system to do what we’ve traditionally done, and it’s what I would call a mediocre middle. It both misses the point of improving education and also misses the point of utilizing the technology, but it’s what we do. My fear is that there has been a financial success in doing things in this way, or at least creating a media culture that equates formal education to the lecture. So you think about a TED talk, or you think about a Coursera lecture… this idea that it is a faculty members responsibility to share their wisdom as the person who’s speaking through it. A podcast is another space, we’re people who are talking in a space. Now that doesn’t mean there’s not a space for podcasts and there’s not a space for lecture, but it’s easy to package that content and put it into a learning space that you’re hoping to monetize. For learning to be effective online and bring down costs, probably requires a pretty seismic shift in how we think about business as normal. Some of the early critiques of online education were that it would turn us into a fordist space, where it was gonna be the assembly line production. That was gonna get away from a faculty member as kind of an auteur, somebody who has the course from its implementation to its full assessment. With online that’s almost impossible to do for the sanity of anybody. So, in some cases, that model is going to need to change in order to be successful. We haven’t figured out what that looks like yet and the human capital costs of doing it right so far outweigh the benefits that you get from allowing students to be able to take classes from a distance and increasing your enrollment, hopefully through online. We haven’t figured out how to weigh the human labor that goes into that. And I think some of it is also we haven’t changed… I’m gonna get radical here, the expectation of what it means to be a professor is still the same as it was 50, 60 years ago, but what we consider is knowledge has changed pretty significantly with Ernest Boyer’s thoughts on scholarship. What it expects to be a faculty member… so the expectations of teaching at even teaching heavy institutions have gone up but the expectations on scholarship or service have not changed. So instead of it being a triangle of scholarship teaching and service it’s this odd triangle that is morphed into a parallelogram with no extra time given to these spaces. So, we’re gonna have to think about our governance structures and our infrastructure if we’re going to be successful. There’s an article in The New York Times this week we’re recording this in mid-September talking about what the next financial bubble may be, and it points to student loans that the cost of education has gone up fourfold over the last 30 years, outpacing everything, including healthcare… and the student loan debt has over the last five years, overtaking credit card debt. It’s the largest amount of debt that exists in any industry. That cannot keep up. Y et costs continue to rise. So another thing; in the next 7 years, that traditional college age, students 18 to 22, is going to decrease in 2025 because of demographic shifts. So, there’s a lot that’s going on at this point, and John, you mentioned the steam engine and how it took decades… Well, we keep saying we’re the Wild West and we only have years until we get to the cliff, and many people would say we’re already past that point; we’re at the point of no return. I like to be a little more optimistic than that.

Rebecca: I’m gonna go back to a little bit of discussion about access. Some of the things that I hear you describing is that the technology is allowing us to have access to information or the distribution of information. Which is why the lectures, the podcasts, et cetera are easy to package and deliver the access to that information. But, what I’m not hearing is access to learning or the access to becoming a scholar, or a way of thinking or being in the world. And I wonder if some of the movements in OER or the open education resources are trying to push the envelope or push the technology and access more in that direction, or if it’s really still emphasizing the ability to just deliver information.

Rolin: Rebecca, you bring up a really great point. And I’ll touch on OER because it’s a fascinating case study in this space, but if you look at the history of distance education with technology, the focus was on bringing people together… that the content operability was not the key point… but it was being able to bring people from disparate geographies or cultures or climates together to learn. And so it is based in constructivism and constructionism and social learning theory and activity theory and all of the wonderful progressive learning theory that is moving teaching and learning today. And the technology that is predominantly used stands much more didactic, maybe behaviorist, in approach because it’s easier to measure that than it is to measure the much more engaging work that happens when you bring people together. So I had an opportunity (I’ll try and not give away any disclosing information on this), but I had an opportunity to work with a group on a MOOC in after the first wave of MOOCs—this was 2013–2014. They were on a major platform and they had created a course, and it was not a traditional STEM course; this was an arts-based course that they had created. And the platform came to them at the end and said, here’s what happened in your class and had this ream of analytics and they said “Well, wait a second. We had a Facebook group, we had meetups, we had a lot of people create artifacts. Where does that fit into this?” And the platform just kind of shrugged their shoulders and said, I don’t know. We can tell you how long someone watched the video and they were saying, “That’s not what’s important to us. What’s important is what were the conversations that were happening and how is that gonna relate to where they’re going further.” We’re in a time of measurement today, yet our measurement structures are much more basic than our capabilities with technology. And so we’re engineering the technology to perfect those measurement techniques. We can’t do much more with bringing people together and engaging more progressive emergent learning theory with technology. I think what George and I were arguing is the technology, as it stands today, doesn’t feed that because that’s not what’s getting the clicks, that’s not what’s moving the needle, whatever metaphor that you want to use in that space. MOOCs are a fascinating space to look at this because the MOOC acronym actually comes from an experiment in social connectivist learning from 2008 with George Siemens, Stephen Downes, Dave Cormier and the great Canadian contingent. And then Sebastian Thrun didn’t even talk about it when he became the father of the MOOC in 2011. He was looking at a bold experiment in distributed learning at Stanford. It was a New York Times reporter Tamar Lewin who made the link between what George Siemens had done and what Sebastian Thrun had done and called it a MOOC. And it kind of stuck and that’s where we went with that. So it’s very interesting to look at the hype versus the research and why the hype is what’s pushing the cart when in academia we like to say it’s the research that does. Now you mentioned OER. I want to focus on that because this is a really fascinating space that in the last couple years you’ve seen this remarkable push on open educational resources, open textbooks, and I am a longtime advocate of open education… been attending the open ed conferences that David Wiley has been putting on since 2013. I ran the unconference there last year. So I’m advocate for what they’re doing. But it is interesting to think about their success and what their advertising is. Their paramount success is really focused in textbooks. So while you have the opportunity to edit a textbook and you have the opportunity for a faculty member to build artifacts of knowledge with students and cross collaboration, that’s not what’s moving the conversation today. What’s moving the conversation are these static textbooks that bring costs down for students. And I like to be the voice that’s saying, don’t forget about these places, because I worry that we’ll see something, and you can even see a little bit of it happening now with publishers who are wanting to open wash or green wash or astroturf what open is and say, “Oh, you know, here we are over here at Pearson or McGraw Hill and this is our contribution to the space.” When you look under the hood it looks remarkably different, but if the focus remains on this static text book in that adoption, it’s easy for that to co-opt. So, to answer the question in a more broad sense, I think in general we have research that’s telling us one thing and we have marketing and public relations and cultural ideology that’s saying something else. I don’t want to say we’ve done a poor job, but the two are very incongruent right now and usually it’s that media PR machine that’s pushing things and we’re playing catch-up and it’s easy to lose track of the research in that.

Rebecca: As a public institution like we are, obviously access as in all people should have access to the information is really important, but I always get concerned about the people who are generating the technology pushing it in the wrong direction and people who value everybody having education and learning not being able to push the envelope or push the technology in the direction that we want to push it in. They’re kind of butting heads in some ways.

Rolin: I would absolutely agree with that. And accessibility, it’s really wonderful to see accessibility being brought forward in terms not only of contents but also of learners, and so the stigmatism of having learning disability or an emotional or physical or some need to engage with content, that now is going to be supplemented by an institution. And that we are designing with that in mind. We’re designing a universal access and UDL that we’re engaging in this space, and that’s a really wonderful change that has happened in higher education. When people talk about the cost of higher education, it’s important to note that things like that are bringing the cost up, and I don’t think any of us would want to get rid of any of those pieces. The problem, of course, becomes “What is the historical understanding of this place?” and “What is our institutional objective and our institutional memory versus these changes that are happening in how we think about teaching and learning?” And I’ve done as much as I can locally at Seattle Pacific University to start conversations and meet people where they are and I think we’ve had some some pretty remarkable success in rethinking some of our structures, but we’re a private liberal arts institution not dealing with the state bureaucracies, not dealing with a state system, not dealing with tens of thousands of students, and it becomes difficult to navigate all of that. Bureaucracy is the least worst tool that we have in order to work with that. But it’s also a great straw man or easy fall guy for any problems that come up, and too often problems continue to exist rather than being tackled because it’s tough to think about what the benefit would be going forward.

John: In your article, you talked a bit about the increasing reliance on private vendors, outsourcing tasks from institutions to vendors on the grounds that that opens things up to the free market in some way, but when we look at the provision of most of these platforms, it’s a fairly unstable market. We’re seeing so much concentration in the market where many small publishers have disappeared, and many of the innovative educational technology providers have been bought up by other large firms. We’ve seen many providers disappear.

Rolin: I’m glad you bring that up, John, because if you think back… and John, you and I have a background in K-12 and it’s really fascinating to think about this from a K-12 perspective because in the 60s and 70s, the heyday of educational film, it was the job of the media resource specialists at a K-12 library to work with faculty to be able to understand how these pieces fit together, and so they were working with Encyclopedia Britannica and World Book and Disney and ABC and NBC and the different content providers of the time, who were making educational titles. It’s a fascinating, fascinating time. The computing craze in the 1980s came at the same time as a recession. And the idea being for people to think about how this was all going to fit together. When this was created the idea was that that role was going to be vital, and the change that happened was we got rid of the media resource specialist and believed it will be up to institutions and collaborations to grow this, to make this go further. Educational film died because it became less expensive to make it and the belief was more and more people would make it. What we instead make are lectures and YouTube videos, and there’s value to both but the great expectation that we had on what these contents could be is gone and we’ve lost that. And so there’s an opportunity… I think that if you think about learning as this contextualized and locally defined space… there’s an opportunity to be able to create these contents. But there’s a lot of risk that goes into that. There’s a lot of quality control that we we didn’t necessarily expect. And there are a lot of other costs that came in and so we output to these third-party vendors, hoping that we hit pay dirt with somebody. In many cases those companies are folding regularly or they’re being absorbed into others, the learning management system ANGEL, which was a rather popular system in the early 2000s and early part of this decade got bought out by Blackboard, and a lot of the people who liked ANGEL liked it for the reasons that it wasn’t Blackboard. But to think about, in that perspective, it’s almost impossible today for institutions to take this on their own. There’s just not a return on investment that works for that, so that means you either have to create these partnerships across institutions that historically have been at war with one another, or you invest in the promises of third-party vendor, either a small one that’s telling you what you want but may not be around in a month, or a large one that you have a lot of trust issues… best-case scenario, trust issues on the kind of service you’re going to receive; worst-case scenario, what does it look like what’s happening to your data, what’s happening to your analytics, what’s happening to the ownership of what you’re producing.

John: Going back to ANGEL a little bit, we used to use ANGEL here and in many ways I loved it; it had some really nice features that Blackboard is a ways away from getting. It had automated agents and so forth, but ANGEL was actually created at Indiana University. And one of the problems they had was that in the 2007 recession, state support for Indiana University was cut significantly and they owned this big cash cow that they could sell off… and so we lost a fairly viable provider in large part because we see in general a decline in federal and state support for higher ed and it puts institutions in a difficult bind where they often outsource more and more.

Rolin: Absolutely and ANGEL is a good example of that. You can go into the 60s with Plato. It was a Midwest State school that was doing Plato. I think about Quest Atlantis was another great thing that gets mentioned in all sorts of progressive educational research that was funded by grants and the funding dried up and there was no way to sustain it. The MEK Corporation, the people who created Oregon Trail and super munchers and that educational software, where is that today? And I work in educational film, I think about it from that perspective. How have we lost those film providers and now we just think that content will fit in for what was historically this really rich and vibrant place to engage, but we’ve lost it on the software side and the teaching and learning side, and we’re outsourcing so much of what we already do to the free market. Certainly there is benefit to that, but at what cost? And I don’t think there’s been enough analysis of what that cost has been.

Rebecca: So, you’re really bringing up the idea that EdTech is not neutral and that there’s competing goals. So, technology companies are obviously trying to make money and then we’re trying to have students learn, ideally. How do we help those things become more aligned? What needs to happen so that we’re not at odds but that we actually find alignment and essentially make the world better which, in theory or in PR, is what’s being said?

Rolin: I think for the first piece, Rebecca, is understanding that EdTech is not neutral, and once we have that foundation, that we understand what we’re using and what it relates to, we can be much more thoughtful about how we use it. So, I am a faculty member but I am primarily an administrator and I use our learning management system here on campus. I could go off the grid; I could try and do something completely different, but it’s important to show support of what we’re doing with an understanding of how that works, and so we have our LTI, whether it’s anti-plagiarism software or proctoring software and all these pieces, and as a scholar I can have criticism of that. So, as a practitioner, how do I help my students understand what they’re getting into with this and making informed decisions about that space. So, I think it really comes into this idea of understanding the learning environment and what my job is: to control… to create pathways for students to be able to learn and to scaffold that and to fill knowledge gaps and help people expand their zones of proximal development, to go Vygotsky on us. I need to cede some of the “management control” that goes into: “Well, we use this, and this is what’s going on.” But, let people make thoughtful decisions about what’s happening with the technology that they’re using. My son in K-12 can opt out of state standardized testing and that’s a decision that’s made as a family. Dealing with college students, we don’t give them the same rights to opt out of some of the technologies that are being used. So, I think about the proctoring technology that was out of Rutgers that was running in the background on computers using retinal scans to engage people and that’s just what you get when you sign up and there’s no informed decision or consent. There’s not even a Terms of Service that you have to read through and then click a button that you don’t actually end up reading. Can we have more of these conversations? Can we be more informed? Because, if we have that information, we’ll be much more thoughtful in the decisions we make on what vendors we choose. The vendors will then have to respond to that market in making software that is more open or more transparent in its use and the application of its data. People have to make a profit. Education has to make a profit. We can marry those pieces together and have a somewhat vibrant marketplace that is serving the learning of students. I think the issue is, right now for EdTech, the student is the customer, not the buyer, and so there’s a gap there that if we have students much more involved in all aspects of that and involved in those conversations, that becomes part of learning experience. I think that that could see some more direct improvements than just generally saying, “well, we’re thinking about this and we’ll continue to think about this going forward.”

Rebecca: I think one of the things I’m sorry I was gonna pick up on the threat of audience but okay

John: You mentioned keeping students in the zone of proximal development. One concern with standard lecture based teaching is that students are pretty much forced to move along at one pace. What’s your reaction to adaptive learning? Is that something that could help, or are there some limitations that we should be concerned with there?

Rolin: I mentioned Plato earlier, the first personalized learning network—basically adaptive learning. I think that there’s a wonderful opportunity for adaptive learning platforms and for being able to bring in competency-based education into spaces. Thomas Edison University has an amazing program that is built on the idea of competency-based education. Alternative pathways and moving away from “seat time…” there’s definitely viability for that. It just has to be thoughtfully executed… and what is the purpose of the learning that is happening in that space? So, if I think about a School of Health Sciences, I think about nursing… if I’m going to get a degree in nursing, there are really specific things that I need to do. I need to pass very specific exams that are proctored in very specific ways that expect me to maneuver in very specific fashions. The seat time is important for that, and that space there needs to model what I’m going to be getting into in an industry. So, I can’t be an intrepid change agent saying, “No, this needs to be social learning theory,” it needs to be what takes off in nursing. No, nursing students need to be able to be successful in the expectations of their field. There are the places that adaptive learning can fit into that. You see it in foreign language in many cases and the supplements that are happening there. Keyboard instruction is another one where that comes in. So, how could we use the best of that to be getting into other spaces. I think some things that we could explore there, as we rethink disciplines and what works for economics or film studies or education. I think there’s some places with that critical thinking… that soft skills, 21st century learner stuff… where the adaptive learning could come in…. so, misinformation, media literacy, fake news… big hot topic and I wrote an article in 2017 that got a lot of attention (not all positive) saying that fake news wasn’t the problem; it’s not what’s ended up resulting in Brexit or the results of the 2016 election. But it was a small part of a landscape that had been neglected and was suffering from blight for a long time because of how we teach this stuff. And I wonder if thinking about digital literacy, which we’re all expected to incorporate into our classrooms, if that could be served by an adaptive learning platform that engaged content, theory, criticism and evocative video to be able to move somebody on a pathway. That’s a place where all of us could come together because there’s no discipline that owns information literacy. It’s built out of information literacy in libraries. But librarians often are the most flexible in thinking about how their craft is going to change. Places like that, critical thinking, the stuff that we’re all told needs to be imparted to our students, but it’s just kind of this hooray concept of “Oh yeah, let’s have this.” Maybe those are the places to really focus on the successes of that and then the research can help define how economics could engage adaptive learning or film studies or education or cell biology.

Rebecca: One of the things that you said earlier is that students aren’t the audience of or aren’t the buyers of the technology. And I wanted to shift that a little bit to thinking about audience and who things are designed to and I think you’re right in that tech companies are selling to administrators who are the ones that are doing the buying and the purchasing who are trying to facilitate certain things, keep cost down, et cetera. How do we shift that conversation so that tech companies start to see the end users who are really students and faculty as the audience of their marketing, of their conversations, and actually shift things so that they focus on the research around learning and improve learning rather than just facilitating something?

Rolin: The key part of what you said, Rebecca, you kept going back to learning, and I think that’s what’s missing in these vendor conversations. We have this idea of what learning is and if I’m a vendor and have mounds of data I can point to achievement and I can point to the things that I measure in my platform that lead to that achievement, and for most instructors that’s not evidence of learning. That might be a small part of it but there’s a much larger picture. And we do a poor job of amplifying that research. That research doesn’t play well in mainstream media, so how do we do a better job of sharing that research. What constitutes learning? What makes learning happen? I love going to YouTube and looking up “do-it-yourself how to fold a fitted sheet,” ‘cause I don’t do a good job of folding a fitted sheet. And I’ve tried numerous times and I still struggle, so that video isn’t the piece that I need to be able to move me there. Now, there are other pieces, potentially making something for dinner that I would be able to replicate in that space, but replication again is not learning. So, even an understanding of: What is learning? What does that mean? How do we define what it is to have learned something? What it is to be a learner? “Lifelong learner” is a commodified term at this point when it really should be a state of being for, I would say, pretty much anybody. How do we engage those conversations? That’s a really complex question. In terms of an institution, how do we bring more student voices into these spaces? and not in a placating fashion of, “Well, we now have a student sitting on this committee.” But to really understand how that student can canvass and caucus with their peers to be able to provide us information. In the same way that if I’m serving on a faculty committee so that I’m meeting my service requirements, but if I’m getting something out of that and I’m giving back that’s a wonderful experience. A student serving on a committee… how can we provide them what they need for their CV or for their graduation in a way that what we’re asking from them they can provide us? …and not just sitting there and saying we’re listening to what they’re providing but often not doing that. So, more student voices in those decision-making processes… more research that’s going to be shown to the vendors… and I think we need to be more thoughtful about those vendor conversations. One thing we do here at Seattle Pacific University, we actually have… with our faculty… we provide entry points for vendor assessment when we do test demos. What are some of the things that faculty who are very interested in being part of these conversations but are coming in the middle of it… what’s happened so far and what are questions they can ask we’ll be able to draw out their expertise and what we need from the vendor? The more of that that we do, the better. I know the California State University systems doing something similar on automating a great deal of the pre-production that goes into assessing vendors so that the stakeholders who are asked these questions have that information in a repository and can access it very easily to make an informed decision, rather than it being brought down from higher administrators… lots of information that’s tough to digest in a small period of time.

John: What do you see as some of the most promising areas where EdTech has some potential?

Rolin: Excellent. My wife loves to say it’s very easy to show why you’re against something, but you get into this business to be for something. Get in education to really share the diffusion of knowledge and help people rise to heights they didn’t know were possible. Fall in love with things they don’t yet know exist as Dr. Gary Stager would say. So, what are some of the positive things that are happening? I really think there’s a chance for a revolution in multimedia. Here’s this podcast that is happening in an interdisciplinary fashion in SUNY Oswego bringing in a faculty member from a completely different perspective who serves an administrator having this conversation. More and more of this is happening. Before we went on the air, John, you were talking about editing your two channels and making sure the sound was right and all of these skills that were picked up that don’t come when you get your PhD in economics. So, as these pieces are coming in how do we value that and so you see more administrations and more governance bodies that are providing value to that. We were talking, Rebecca, about open education. The University of British Columbia now will recognize the editing of OER materials as part of promotion, tenure, and review for their School of Education. That’s a phenomenal change that has happened in how we think about the role of the faculty member as a distributor and conveyor of knowledge. I think people are being more thoughtful at this in this day and age. But, you did ask specifically about technology, so I need to pivot back there for a second. I love some of the stuff that’s happening in virtual and augmented reality. Some of the really interesting research that’s happening there. I like the drop-in classes that are happening around special interest topics that often, in many cases, are informal or non-formal learning spaces. Museums putting on areas where you can come in and learn in a certain time. Kind of a gap between a human experience and the MOOC but you’re kind of doing both at the same time. I think that the opportunities that we have with free and ubiquitous devices… and I don’t mean free as in cost but I mean free as in access to… and especially in the West with broadband capabilities, what’s going on with video and how we can better engage that and as more people learn about nonlinear editing and cinematography and camera and sound. What are some of the resources we’re gonna build there? Opportunities for students to share their knowledge is the main thing that comes forward for me. WordPress, which runs, what, a quarter of websites in the world is getting incorporated more and more into courses. You think about the WordPress camps. There’s a great thing happening in New York City coming up on managing the web and how you can work with students to be able to be creators and owners of the knowledge that they’ve created and what the implications are in that space. It is kind of a tough time to be bullish on technology if you think about Facebook and Google and Apple and Amazon and antitrust that’s going on in all of those spaces. And so a lot of the stuff that I’ve mentioned here is somewhat renegade, somewhat guerrilla even. So where are those opportunities to engage with environments through online? It comes back to community in that space. How do you find and foster that in your networked identity. There are opportunities and more and more that’s going to be happening. I think that we’re in this storm and after this there will be, not a calm, but there will be an opportunity to look at what’s been broken and how can we build and improve going forward, and I think that we’re getting to that point sooner rather than later.

Rebecca: We generally wrap up by asking what’s next. You talked a little bit about what’s next in EdTech, but what’s next for you?

Rolin: What we’re doing at Seattle Pacific University around academic innovation; we have been offering seed grants to faculty for the innovations that they see as necessary, whether that’s in a classroom, in a department, in a college across the entire campus working out in the community. We provided 45 of those over a two-year period, so almost a third of our faculty directly affected by those and it was very powerful, so we’re taking that a step further and engaging at a school or college level and finding innovations that we can then potentially put into day-to-day operations. So, one of the things we’re thinking about actually are adaptive courses. What would it look like for a course in nonlinear video editing to be almost entirely online. And you think about that with lynda.com. I can go to lynda.com and take a tutorial in using Final Cut or Adobe Premiere. What am I getting out of being in a higher education institution that I can’t get off of Lynda? That’s what we’re exploring: what does it look like to have that scaffolding and support that’s directed toward a greater understanding of knowledge? Other things are definitely around social justice. We are seeing at Seattle Pacific an increase in first-generation and historically underrepresented students who are coming in with the same scores as their peers but, once they get here, we’re seeing a discrepancy between where we would expect them to score and where they are scoring. And we have statistically significant research showing that that is the first-generation student demographic. So, what are some pieces we can put into play to be able to help them with their success? Because it’s not a matter of not being able to do it; it’s a matter of the structure and the culture is not befitting them. So, we have a program called the Bio Core Scholars where we are working with tutoring and mentorship on research, community, and knowledge gaps to be able to move these students. We’re in our fifth year of this program, we’re looking at expanding it. But we have brought the students up a full standard deviation in their scores, and we had an 86 percent success rate in graduating people to pre-professional health programs, which is just a remarkable number. Personally, I’m really big on what we can do with educational video. What are some of the things instead of it just being a lecture? I love Skunk Bear on NPR, taking a topic and in three minutes doing an entertaining, evocative dive into that topic, but again, that’s Oliver Gaycken would call “decontextualized curiosity.” How do we take that and actually put it towards learning? So, I’m looking at what does it look like to have lecture mixed with a very product based assessment mixed with more evocative filmmaking to move people into learning? How does that all go forward? It’s a very exciting time to be in higher education, even with all of the things that are looming on the horizon.

Rebecca: Certainly doesn’t sound like you’re gonna be bored any time soon.

Rolin: Not at all.

John: Thank you for joining us. We look forward to hearing more about this.

Rolin: John, Rebecca, thank you guys for having me.

Rebecca: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Theme music by Michael Gary Brewer.

30. Adaptive Learning

Do your students arrive in your classes with diverse educational backgrounds? Does a one-size-fits-all instructional strategy leave some students struggling and others bored? Charles Dziuban joins us in this episode to discuss how adaptive learning systems can help provide all of our students with a personalized educational path that is based on their own individual needs.

Show Notes

In order of appearance:

Transcript

John: Do your students arrive in your classes with diverse educational needs? Does a one-size-fits-all instructional strategy leave some students struggling and others bored? In this episode, we examine how adaptive learning systems can help provide all of your students with a personalized educational path that is based on their own individual needs.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guest today is Charles Dziuban. Chuck is the Director of the Research Initiative for Teaching Effectiveness at the University of Central Florida, where he has been a faculty member since 1970 teaching research design and statistics. He’s also a founding Director of the University’s Faculty Center for Teaching and Learning. Welcome, Chuck.

Chuck: Oh, thank you both. I’m just so pleased to be here. The good feelings of Oswego continue… you know my growing up upstate New York and my attending Oswego. I’m really honored to do this… makes me feel good.

Rebecca: So glad to have you.

John: We’re very glad to have you.

Rebecca: Today our teas are:

Chuck: I have water from the water fountain.

John: My tea is Yorkshire Gold.

Rebecca: I have Lady Grey today.

John: We’ve invited you to join us to talk about adaptive learning. So, what is adaptive learning? Is it something new?

Chuck: Sure I’d be happy to talk about that. Let me answer that question in reverse order. It’s anything but new. I think adaptive learning has been around since we’re thinking about teaching and learning. I think if we went back to the Middle Ages we have an idea of this notion of adaptive learning. It is not new. In fact, where I came from, and what I’m in my career, I’m looking at a paper written by John Carroll called “A Model of School Learning…” where John laid out this set of equations that basically answers this kind of a proposition: if how much time a student spends learning is constant, what they will learn will be the variable; if what they learn is the constant, how much time they spend in the educational enterprise will be the variable. So that’s the basic sort of notion of that. When we define that, it kind of makes sense. If we give students a limited amount of time and anything… a course… a university… a semester… whatever you call it. It makes sense that what they acquire in that activity will be the variable. If what they learn is a constant, how long they’re going to spend in this enterprise will be the variable and we’ve all experienced that, but we’ve all butted our heads up against something for years and years and years until we finally understood it… and this is the basic notion of adaptive learning…. whatever time it takes, it takes for you to learn something… and he went further to break that down. When we say time, it’s just not how much time in the Augustan sense, it’s how much time do I spend actually in the learning enterprise and what is your perseverance… and what is your aptitude for this… so kind of modified for that. So, that makes eminent sense. That’s what adaptive learning is… basically it’s going to take whatever time you need as a student to learn this material to some sort of satisfaction level. That’s what it is. It’s a very simple concept.

John: …because students come in with very different backgrounds. Some have very rich backgrounds in some areas and weaker in others… and that’s going to vary quite a bit depending on what they’ve learned in prior courses or in their life experiences.

Chuck: Oh, absolutely. I mean I can remember my own experience walking in new courses at Oswego being virtually lost because I had no background and no experience in the course and having to really ramp up my energy and effort and use of time in order to: 1. reach the baseline in the course and go on in other courses. In adaptive learning we have students who enter courses who… don’t take this the wrong way… but fundamentally don’t need to be there. They’ve mastered the material… they can move on… and I’ll give you examples of that later. So, this gives a great deal of flexibility to the learning enterprise.

Rebecca: From the learning perspective, it sounds really wonderful. From a teaching perspective, it seems like it can be a little bit challenging – especially before some of these technologies were available.

Chuck: Nightmare. Yeah, it is. Of course, the way we’ve conceptualized this whole notion of how students learn, it really does… We’ve organized learning around this entity called the class… it’s a unit… yeah, and we call it a class… and if you think about it in terms of a course sequence, say in mathematics, the students who don’t pass their placement exam so they take intermediate algebra, and college algebra, and trig, then precalculus. You can conceive of that as these discrete units, but if you think about it, what we’re doing in our math department… we’re designing that as a series of skills across those things we’ve called classes and that makes the whole notion of teaching quite different. Really what you have to do at adaptive learning… if you begin to plan a course you really have to look at what you have taught, what the granularity of the course is, and your role as an instructor in all of that. It’s very very different and it really forces our instructors to think about what it is I’m teaching… it is much more than a syllabus. The granularity of what you teach becomes critically important.

Rebecca: How did UCF get involved in adaptive learning?

Chuck: It’s an interesting question. That was organic. You may or may not know that UCF is the fastest growing university in the country. It is one of the largest universities in the country. The good news is we’re growing; the bad news is we’re growing. [LAUGHTER] We cannot possibly build an infrastructure to house our students. So, besides adaptive learning, we have online courses, we have blended courses, we have lecture capture courses, and adaptive learning is a natural outgrowth of that sort of structure. 43% of our full-time credit hours are produced online so we’re almost a half online university… and one of the things is we became involved in this simply because one of the things that John said. Our students are very diverse. They come with very different backgrounds and even putting them in this thing called a course may not make sense for some of them in terms of their pre assessment of where they are… and the notion of adaptive learning pre-assessing them and then putting them in the proper learning sequence within a set of objectives is why we got involved in adaptive learning… to doing this… and what we did is we simply recruited our best faculty, say “Look, see if you can do this. See if you can, then kind of accommodate what students are, where they are, and how they’re learning, in a much more flexible environment. It just simply make sense for students… and given the fact that we have a diverse background…. students coming from underrepresented populations… it just makes sense at so many levels.

John: One case where we’ve done that institutionally here at Oswego is, beginning about three or four years ago, we adopted Aleks for the math requirement. It’s used for initial placement and then it’s also used for students to get up to the level needed to meet basic course requirements. So, students come in and then if they don’t meet the requirements for the courses they’d like to take, they can spend the summer working on that to move up to the level… and then test themselves again to basically get to the level they need to be to make good progress… and it’s been working very well from everything I’ve heard, but we haven’t done much more yet at an institutional level.

Chuck: Yeah, well there are, as you probably know, with technologies like this there’s a probably a new platform coming out every day. I mean this as a metaphor. Our math department uses Aleks and if you go into an Aleks class… and Aleks is a pre-designed course, there is some flexibility but it’s a course that’s pre-designed. These platforms come in basically two varieties: where courses are embedded in them like stuff with McGraw-Hill or Knewton or something like this, or content agnostic like Realizeit where you have to build the course. That’s two kinds of things. We have an Aleks course in college algebra, and if you look at it, it kind of looks like chaos. Students are coming in, there are people lecturing on the boards, there are student assistants helping them, they’re on the thing. When they master it, they master it and they leave. So, any concept of a lecture course is completely out the window, but what’s happening is: students are being assessed, they’re being reassigned to different areas within the Aleks platform to master the skills, and then moving on. So, some of them can complete a course in half a semester… and I assume you’re experienced at seeing this at Oswego, but one of the things we’re facing is where with adaptive learning, we’re trying to get over the teach then test model for a course. We’re trying to embed the learning assessment within the adaptive learning platform. So, we have a test-free course where essentially assessment becomes part of the learning platform assessment… actually becomes part of the teaching… and it’s really quite exciting, but it is very daunting to do. We’re kind of wedded to tests in our environment.

John: I believe that you’ve been working with Colorado Technical University.

Chuck: Yeah!

John: How did that partnership come about?

Chuck: Organically.

[LAUGHTER]

When we came to realize that adaptive learning may offer some advantages for us, we asked several vendors to come in and make presentations to faculty… and this was basically a faculty decision. They saw several platforms and they said “Realizeit is the one. This is the one… because we want the flexibility to build our own courses.” But, of course, there’s a Chinese proverb: “Be careful what you ask for….” and they got Realizeit… but now they had to build the courses. That’s daunting when you start building your courses. Realizeit’s been very helpful. They have a process called ingestion where they’ll try to take whatever you have in a course… and I’ll say this and I say this with love and kindness… all of these platforms have a bit of clunk associated with them. They all work, but they are all some problems associated with them and problems in quotes. I’ll say challenges, or in Provost speak, they have opportunities associated with them. [LAUGHTER] …and we work with them… but in doing this kind of thing, but we began presenting this with our partner (and I’ll talk about our research relationship with RealizeIt in a little bit)… but working with RealizeIt in some research that we began presenting and Colorado Technical University has been doing something that we’re not able to do yet, that is scale it. Scaling it is an issue. What they have done is, they’re a private for-profit institution, and they’ve scaled it at a remarkable kind of thing and thrown a lot of resources at it. So, we’re very good at research, they’re very good at scaling, and Realizeit is very good at research. So, when we saw them presenting and they heard us presenting, it was sort of a speed date. They said “hey, you’re doing pretty cool stuff” and we said to them “hey, you’re doing pretty cool stuff, you want to play?” …and that’s how it started… working with them and it’s led to several publications and several presentations, and then with RealizeIt looped into this, working in the background, we’ve been able to do this kind of thing where we think partnerships are really important in studying this… and of course I’m gonna recruit Oswego today. You should be joining us.

John: I’ve been looking at adaptive learning platforms for a few years.

Chuck: Yeah.

John: I’ve worked with some from the publishers but it would really be nice to build something from the ground up. So, we’ll talk about that perhaps a little bit more later.

Chuck: Alright. Well, I will say this. Our philosophy is we give away everything. You can have anything we have. We’ve published several papers. We have several projects underway now with adaptive learning. Any of your audience, and certainly my brothers and sisters at Oswego can have anything UCF has done. The idea is we need to do it for partnership. I did a presentation on, if you go to Google and search “the Grand Cafe.” The Grand Cafe was the first coffee house in England, and what it was was a result when the British discovered coffee houses ideas really blossomed. The Brits didn’t drink water for a long time because they were afraid of the water, and so for a couple of hundred years they basically started having a beer for breakfast.

John: Right.

Chuck: A couple of beers at mid break, and then a beer and a gin. They spent a couple hundred years drunk.

John: ….and so did the founding fathers.

Chuck: Yeah, absolutely. But when they discovered coffee houses… a coffee and tea (which you’re drinking)… they were in this space they were in this learning space and when you switch from a depressant to a stimulant there are many more ideas and that’s the grand Enlightenment… but the point being that it was a partnership. When we talk about these grand ideas they are not Eureka moments, they are not lightbulb moments, they are people working together creating ideas and, if you will, letting ideas have sex… in the idea that these ideas can grow and be developed and that’s what happened…. and that’s what happened with our partnership and now we’ve added another partner. We’ve added a Petroleum Geo-Services in Oslo, Norway. It’s an exploration company looking for oil all through the world, and suddenly they have to train these people aboard ship and they looked at this and said: “Whoa, it costs us a fortune to bring these geophysicists back to Oslo to train them. I wonder if we can do it aboard ship with adaptive learning.” So, we’re looking at that. So, there are a lot of possibilities and I’ll say we worried about the students adapting to adaptive learning. No problem. Of course, it’s embedded in our LMS so they don’t even know they’re not in Canvas. They still think they’re in Canvas.

Rebecca: So, you alluded a little bit to Realizeit’s potential, because you mentioned that it’s one of the few platforms where you can build courses from the ground up. Can you expand a little bit more about its strengths relative to some other platforms and maybe also some of the stumbling blocks associated with building something from the ground up?

Chuck: I’m not sure I know the characteristics of all the platforms. I know that Realizeit is content agnostic, and I’m sure there are other platforms that allow you to do that… and probably there are other platforms available too, as you begin to explore this where courses are assembled but they have some flexibility for faculty members to take components in and out. I would think that would be a real advantage. It’s sort of the continuum… building of course from ground up is daunting and one of the problems is the granularity of the course. Getting this level of granularity so the course flows evenly…. it’s always a rough start in terms of doing that and we’ve provided as much support we can for faculty.

John: How can faculty get started using adaptive learning? Is this something that’s best done at an individual level, or at a departmental or an institutional level?

Chuck: It’s a big task and whatever support you could provide… especially instructional design support… in terms of looking at a course, redesigning it, and putting it together, and that’s very important… and run-throughs are very important, test beds for courses before you roll them out. I, in my role having to evaluate our technology, I taught online, blended, lecture capture, adaptive, and each time I rolled it out… it’s been terrible. It has a lot of bumps along the road and our working two years with adaptive learning…. they’ve had some issues and by our work with them for two years, we have helped them clean up a lot of issues… and they would be the first to admit that. They help out on the research side of this. They’re very very strong. They can do research things we can’t. We can do research things they can’t do. It’s the perfect yin and yang… and this is a wonderful kind of relationship to have…. and Michael Feldstein and Phil Hill and the MindWires e-literate have built this project now they call the Empirical Educator where they’re trying to get vendors and academics together to work together… to begin to look at this.

John: We will include links to the articles that you’ve mentioned in the show notes.

Chuck: Absolutely. Okay.

John: The question I had about that (in terms of individual development or other forms of development) was basically whether people are developing their own variants of the course or is it done at the departmental level? Or the program level?

Chuck: Both.

John: Okay.

Chuck: How do we begin an adaptive learning? One of the things you have to do to demonstrate to faculty that it can be successful. So what we have done is we cherry-picked our best faculty, the kind of faculty that if you show them this platform, they’d say, “hey, we’d like to try this.” So our initial pilot study was done in Psychology, was done in College Algebra and Pathophysiology. And all of these faculty are the ones who would say, “I would try to do this anyway,” and we built it that way and then began demonstrating it with individual faculty. And now we have a pilot project in our College of Business to make as many courses as we can adaptive learning. So it’s now up to the college level and we are now making presentations to all of the individual departments saying, “look, this is the possibility— is this a strategic initiative for your department? Do you think this would be something that would be a value to a department? A value add to your department versus one-off courses?” Because, for instance, in Psychology, the Psychology professor developed it over three semesters and faculty really do have the option of how much they can lock down this course. You can lock down an adaptive learning course so it looks like a regular course or you could make it go completely adaptive. It’s pretty scary stuff and what Jeff did in Psychology is his third go-around, he said, “it’s adaptive, go!” That’s all there was to it. They could go through at any pace they wanted to. In a cohort of say 20% finished the course in three weeks and that sounds like heresy, okay. But they did it and they were finished. Their next thing they said, “okay, we want Psychology too, we’re ready for it” and wasn’t ready. Okay, it wasn’t ready. So what do you do with them— your students who have completed a course in three weeks. They’re done, they’re verified, they’re certified that they’ve completed the course and they have nowhere to go. So it has great consequences for building this out in terms of doing this and we work very hard. Intermediate Algebra (they didn’t make it into College Algebra) so Tammy Muse, who will be featured on 60 Minutes in a couple of weeks, has allowed them to start Intermediate Algebra when they complete [it], go directly to the Adaptive Learning College Algebra course and that cohort now is finishing the course on time and it’s really quite amazing. And what’s the objective is, well, make that cohort larger. Just the things that you were saying about Aleks, (that’s a bad word in Florida, “remediate”), but let them acquire the skills they don’t have and then go directly to the course they need. And what Realizeit will do is sequencing them back and Tammy has done an amazing thing. She has made all of her assessment items reflect the diversity of our campus with names and the diversity of the disciplines she made them have the disciplines reflect whatever their major is, whether it be Engineering or Physics or whatever. And though the problem sets “they work” all reflect those kinds of things. A lot of work, but it’s working beautifully.

Rebecca: Sounds really exciting.

Chuck: Well, on certain days, it is.

Rebecca: [Laughter] Exciting can be both scary and…

Chuck: Oh yeah, yeah yeah, yeah.

Rebecca: We talked a little bit about whether or not the platform allows for interleaved practice. How does it work, what does that look like? Can you describe one of these courses?

Chuck: As best I can.

Rebecca: Yeah.

Chuck: It looks like you’re in an LMS (Learning Management System) and you have exercises. RealizeIt has a decision engine built in and I know a good deal about the decision engine. I don’t know everything about it because these platforms are proprietary. These vendors do not like to give away their trade secrets but Realizeit is Bayesian-based. It gets prior knowledge assessment of a student, then based on that prior knowledge, it assigns a student to a location in a course and begins and then begins to assess them. It looks like an LMS, the learning can be anything from videos, to simulations, to reading, to practice exercises, to discussion boards— it can be any format that you would normally have in any particular course. Students can participate in any number of ways they are assessed and then based on the assessment, which can be (I’ll talk about that in a second), can be anything. It can be a simulation, it can be a practice exercise, it can be a performance— it can be anything. It evaluates them and based on the evaluation, it reassesses their learning path and then their sequence back into that and they can have any number of ways to go through the learning factor. So this platform is always thinking about them but it is re-assigning them and it is constantly re-assigning them to learning passes that goes all the way through. It’s kind of scary to look at in terms of what’s happening and in it then, it has for them, “you’ve mastered these, you have not mastered these kinds of things,” I wish I can tell you it works as smoothly as I just described it…

John: [LAUGHTER] But it’s getting there.

Chuck: It’s getting there and what we tend to do is overestimate the short-term impact of these things and underestimate the long-term impact— that’s something we do with all technologies. If you look at the latest MIT 10 technologies that are gonna revolutionize them, one of them is Babel Fish earbuds that instantly translate languages for you. Well, they can’t do that exactly yet, they will eventually maybe get very good at that, but that’s what it looks like. It looks like a regular LMS and if you experience it, you have this sort of seamless feeling that you’re moving through this with no real impediments to this so you kind of go through your own pace. And what it also does it learns how you best learn. If you do it best with reading, it’ll do it that way. If your best with simulations, it’ll steer you up to simulations, but it’ll steer you to them. But faculty have to prepare all of those things so “therein lies the rub” as a friend of ours in the round theatres in England would have said.

John: I looked at the Acrobatiq platform a couple years ago, I met with some of the representatives from the gardening. I haven’t looked at Realizeit very much yet, but these platforms are really good at giving students lots of retrieval practice and assessing where they have weaknesses and doing that type of [a]daptation. But one of the things we know is that interleague practice, as Rebecca mentioned, is really helpful in increasing recall when you ask students to go back and test them on things that they learned earlier in the course is really helpful and encouraging deeper long-term learning. And I asked about Acrobatiq’s ability to do that and there wasn’t any and that’s why we were a little bit curious about whether perhaps Realizeit had that ability to go back and bring in questions earlier. I know Aleks does that a little bit but most of the platform’s I’ve looked at so far haven’t.

Chuck: Well with Realizeit, students have that option. I guess if we could make interleave a verb, they had the option to interleave, I’ll give you an example of that. Pathophysiology— in the state of Florida there’s a requirement, now, most hospitals require their RN’s to become BSN’s and that’s causing some angst. Nursing is stressful enough but now nurses are given a certain timeframe to achieve their BSN’s so nurses are coming back, some of them unhappily. RNs to get their BSN’s and you’re doing it in adaptive learning platforms and online platforms in the Pathophysiology was adaptive so Julie Hinkle, who taught the course, said, “I’ve got 30-year RN’s who worked in Cardiology their whole career coming into my Pathophysiology course, taking the Cardio unit.” Well, they know more about cardio than I ever will and so within the adaptive learning platform, they simply go to the Cardio unit and test out. They’re done, ok, because they know everything but nurses are funny. If they don’t get quote “a hundred,” they’re not satisfied. So that’s their interleaving, they will go back and test themselves again and again and again and again until they get the satisfaction. High pressure, high pressure in that field. It’s not like beginning Psychology— if I pass, I pass in terms of this. And what Julia’s done is created incredible adaptive learning measurement devices. She’ll give them a series of bloodwork in blood gases that they all have to look at. I mean it’s hard and they all get different values and they all have to assess this patient and the protocol for this patient based on different values associated with and then she’s got them in discussion boards and if the values don’t make sense, the nurses, because of who they are, will go back until they get it. They had the option to do that. In this case that even you talked about her student-driven, you don’t have to force them— yeah, they have the option to do it, some will, some won’t.

Rebecca: Have you been using any open education resources as part of the content for the adaptive courses or are these all closed system faculty you’ve created? Materials…

Chuck: Our faculty, when they begin looking at it, you had this sort of adoption curve, we got the early adopters, who will do anything. When we look at our online courses— we’re very good at online teaching, we are very very good at it. However, we probably have 2,000 courses in our vault and faculty will ask me, “will I be any good teaching online?” Well, my response is, “are you any good face-to-face and if you’re not, you’re not gonna be very good online.” But we range from faculty who do things that are very text-based, faculty members who will not stop putting bells and whistles in their courses. And we say to them, “stop it, you’re being annoying. Stop with the gizmos in the course.” Somewhere, there’s a balance in this kind of stuff and obviously what you have for this preparation of course is a lot of work. I don’t know about you and Oswego, I assume everybody teaches one course and has a lot of free time [LAUGHTER]. We’re not that way at UCF. Our courses are very large, heavy teaching loads and right away, faculty say, “this is the too daunting.” So what I realize is doing in a lot of platforms are doing that are like us. We’re looking at OER (Open Educational Resources) and saying, “what of this can we ingest into this and make it available for a course.” In terms of, “can we take some of the right stuff, put it in there, and load it up so it’s ready to go” and then you can adjust it as we go along. Obviously those courses are good and if we can do that for faculty, it’s a great service because building a course ground up is daunting no matter what happened. And unless your institution provide faculty support, it’s probably too daunting for an individual faculty member to do this by themselves in my estimation.

John: Does Realizeit provide any package materials to help get people started, for example, including OER.

Chuck: Yeah they do, they provides a great deal of resources for it. They have said representatives down when things were not working as well as we’d hope. Basically all of these vendors and I’ll say they have the Veg-O-Matic, we have the platform for them to try it and it doesn’t chop vegetables equally well. That’s the partnership that’s so critical and there’s vendors are not just vendors anymore. They have to be active partners in this educational enterprise. They’re not just selling us stuff— they have to come and help us and Realizeit is very good about that. And others are, too. I’m not pitching Realizeit by any stretch of the imagination.

John: But it sounds like it’s worked pretty well for you guys.

Chuck: Yeah, it’s worked pretty well for us. Our issues are, “how do we scale it, how do we scale it?”

John: What sort of resistance has ever been from faculty? Is it mostly just a time issue or there are other issues that faculty are concerned about?

Chuck: I can probably give you a less of them. One: it’s a lot of work. For all faculty, I think one of the questions are, I’m not sure of the culture and Oswego, but at UCF, it still remains get-ahead equals teaching service and research and we know what of those three carries the great weight.

John: Yes, that’s an issue that’s discussed all the time.

Chuck: So if I’m gonna spend all of my time in preparation of this kind of enterprise, where’s the reward system within the culture of the University for this? And I don’t mean to be crass but what’s in it for me? Why should I do this? And we have to provide some sort of reward perfectly. We’ve done a lot, we have sowed awards that rewards faculty thing, but we tried by giving them course releases and all of the help we can have. We have a large staff that support faculty in the online environment and we have a bank of instructional designers that help faculty members load up their adaptive learning course. And basically our philosophy is if faculty can feel better about their teaching, they’ll follow you anywhere. I mean that’s yet— you know we all want to be the best teachers we can and it feels so good when it goes well and it feels so awful when it goes poorly and we’ve all had that experience.

John: Oh yeah.

Chuck: I am personally— I’m very fond of the quotes attributed to Augustine. I thought I understood it until I tried to teach it. Not one of us has not experienced that at one time in our position. [LAUGHTER] I’ve been talking about things that I’ve had no business talking about. I knew them, but I didn’t know how to teach them.

John: But you learn that as you go. [LAUGHTER]

Chuck: Yeah, oh yeah yeah, yeah.

John: In your recent EDUCAUSE paper on Adaptive Learning, you describe different types of student interaction with a platform. Could you tell us a little bit about that? I remember tortoise and hares and a few other animals in there. [LAUGHTER]

Chuck: Yes tortoise, hares…

Rebecca: Frogs and kangaroos.

Chuck: Frogs and kangaroos [LAUGHTER]— I’ve got to tell you, Realizeit is headquartered in Dublin, Ireland so one, it’s fun to listen to them talk. And two, they’re very bright so in doing this, what we decided, and then we’ve done this a lot, we had a research partnership and you’ve probably known this. If you begin presenting some of this stuff in tabular form and tables, your audience glazes over. In about two minutes, they stopped listening. So the idea there is, “how can we begin to portray this data in a way that’s more engaging to audiences?” And working with Realizeit and Colm Howlin, who was their Director of Research— is brilliant. We’re both big fans of Hans Rosling in Sweden, passed recently, but his approach to showing data in motion…

John: Oh, it’s wonderful, yes.

Chuck: It’s just absolutely wonderful. So we thought, “can we look at this” and what we weren’t able to do is portray the Hans Rosling Gapminder way, is looking at students traversing through an adaptive learning course of Psychology that is showing them in motion two things: what they’re achieving versus the number of activities. And that’s available from us in an animated form and then looking at them as they traverse through there, we have found that they produce certain behavioral styles and we found originally four. And the people in Ireland named this— they’re metaphors. Turned out to be animal[s]. The first one was the hare, and the hare— and remember what we’re plotting: number of learning activities against achievement level. The hare just goes zip. They start the course and they finish it in a very few days. That’s what they do— they’re finished. When I described in Psychology, that’s what happened in Psychology— they’re finished. And it sounds like heresy, but they’re finished. They’re done, ‘kay? And then the second one was the tortoise. The tortoise is the one who goes step by step by step by step by step through the course. They just progressed through the course in little increments and do this. And they get there, but they get there slowly. And then the Frog. The Frog did exactly what they would do in a regular course. There were eight modules in the Psychology course— they completed one a week. They did it week by week by week by week the way they would do it in a course. And interesting enough, when Jeff turned the cord, Jeff Cassisi, he turned the course open. There are a lot of students who did that— they went week by week by week because that’s what they’ve been taught. For them, that’s what of course was: you do it in these increments. And then the kangaroo— didn’t do a damn thing until the end of the course and then did it all in three weeks. Did it all— just zipped to the end of it and it’s fun to watch him in animation. He’s just dead in the water until the end and then he goes zip! And he’s at the end of the course. So there are four different ways that they approach it in terms of doing this and to see that in motion is very compelling. What it tells you is, at least in this course, there are various behavioral styles to the course and you have to be comfortable with them because they’ve all reached mastery and they’ve all reached mastery in some ways we wouldn’t approve of it. We wouldn’t approve of that kangaroo, would we? Doing nothing until the end of the course. That’s not right, but they finished so we have to learn how to deal with that in terms of doing this.

Rebecca: Despite the fact that many academics exhibit that exact behavior… [LAUGHTER]

John: That’s why we have deadlines. [LAUGHTER]

Chuck: Yeah, we have deadlines. Yeah, nothing is quite as motivating as fear, right? [LAUGHTER] yeah yeah yeah, but one of the things that I’ll say (is) there’s a lot of research that’s affecting us. And a really compelling book is called Scarcity, I don’t know if you’ve ever seen it, by Mullainathan, where they talk about students completely compelling to us at UCF for students coming from underrepresented populations in terms of what you think of students living close to the poverty line, what are they dealing with. They’re dealing with so many things in their life. Money, time, danger, single parenthood, finances, family, two part-time jobs, borrowing money from college. Fundamentally, when these students come to us, they’re exhibiting and expending so much cognitive energy just living life. Then when they come to us on campus, they burned up most of their cognitive bandwidth and then we put them in these lockstep courses where for some reason, they miss a class, they’re behind the power curve. We don’t design courses for these students. We firmly believe that they have as much intellectual capacity as any other students but we have not designed our University to accommodate. Adaptive learning is perfect for these students, it really is in terms of doing it so we’re trying to do that and accommodate these kinds of things. No wonder they drop out. In some cases, dropping out of a course becomes the optimal decision and what it is is their superstructure of life is so tenuous that if any one thing fails, this whole house of cards come tumbling down for these students. So they get behind a couple of classes and they’re done and we have to find a way to accommodate this. We’ve got to begin adjusting the way we organize ourselves.

John: What implication does adaptive learning have for the structure of the University?

Chuck: It has a lot. Our students are, now that they’re learning, asking really interesting questions like, “why do we need semesters” and the only answer we have is because we have semesters. [LAUGHTER] We don’t really have good answer for that and we’ve done this an adaptive learning we designed it. Now if you finish early, well you go on to the other course. The other end of adaptive learning is so what if you need an extra three weeks to complete the material and the semester is over? But that’s a nightmare for Financial Aid, it’s a nightmare for the Registrar, but we’re working on that in terms of doing this. But it hasn’t potentially turned on University structure on its head and I’m not sure we’re ready for that at the moment.

Rebecca: When a faculty member creates one of these courses, obviously they’re very involved in that content structure, but what’s their role in facilitating the course?

Chuck: Mm-hmm.

Rebecca: So if the class was to extend three weeks past for a particular student, does that mean then that faculty member is also engaged for that three weeks or what does that look like?

Chuck: Absolutely. In for a penny, in for a pound. If you’re involved with this kind of thing, if a student is still working within the platform and we’ve made agreements with the Registrar that they would get credit for the semester in which they did this, although they went past all of the due dates, a faculty member is still involved. So it changes the role of the faculty member immensely in terms of doing it. Some students (this is gonna come out wrong but), don’t need the faculty member. I mean, they need the faculty member, of course, I need the faculty member. This by no means abrogates the role of a faculty member, but it certainly does change it. Faculty members have to know when to intervene, when not to intervene— it changes us. It’s like teaching online: you have difficulty adapting to this. When I taught an online course, I did have difficulty adapting to my role, not being the center of attention all of the time. It was hard I have a big ego and it was difficult for me but but I got over it.

Rebecca: Are the assessments related to the adaptive learning stuff that’s like automated or something that the faculty member is manually grading using rubrics or things like that?

Chuck: Both, they are both. They get up to the faculty member we have a department who is now just getting over the notion of letting the assessments run within the adaptive learning platform and believing the assessments that it is as competency-based and students exhibited competency. That’s a hard sell for departments. Say a math department who say, fine, but they still have to take the test. You know what I’m saying? It’s a slow-moving thing and what we have done now is an a/b study in terms of demonstrating the fact that students who are assessed within the adaptive prep won’t do as well as students who were in other courses and they took the departmental exam and did equally well. We have to demonstrate that. It’s a hard sell. It’s very scary. It is a very scary phenomenon. Yes, you can design all kinds of platforms and you can build all kinds of intermediate testing devices, you can give tests within the platform and if students do well on the test, it can be, automated is the wrong word, it can be Bayesian decision to cycle them back to where they need to exhibit their skills and you can retest them. So it’s this continual cycling kind of thing.

Rebecca: I have a really easy time imagining how this would work for a knowledge gaining sort of class, maybe an a lower level of course, but a harder time envisioning when it might be like in an upper division class or one that might be more project-based or application based. What’s the experience been on your campus in terms of introductory level classes versus upper level classes giving adaptive learning a chance?

Chuck: That’s a good question. Question posted versus how would you teach Macbeth in adaptive learning? How would you do that? How would you teach clinical psychology in an adaptive learning course? The answer is that probably adaptive learning is not equally well suited for all disciplines or all levels. One is, adaptive learning is really really suited very well for hierarchical structured courses where achieving something at one level depends on achieving something at a slightly lower level. Like sequencing in math, or chemistry, or physics, or computer science. You can do it and we get it beginning psychology but is it really necessary? When you have eight modules in beginning psychology and there is a natural organic order but they don’t necessarily depend on each other. Now, Young came after Freud but, is it really dependent? That’s the kind of thing, I think there are some areas where it’s much better suited, and your question is well taken. I think we have to do a lot of exploration in terms of where adaptive learning is most suited and fits into our curriculum and it may not well fit equally well across all disciplines, all ecology. It’s a question we’ve got to do a lot of work on. Hopefully SUNY Oswego will answer most of those questions. [LAUGHTER]

John: Next semester. It’ll take a little while.

Chuck: Yeah, right. Right.

Rebecca: Do any of the classes involve, that you’ve been highlighting on your campus, have writing as a key component? I’m just curious.

Chuck: Yeah. I think probably in psychology there would be some writing involved in it. They would have to do some reaction papers and do that, yeah. So I think I can comfortably answer that question as, yes, there have been. It’s equally possible. The question is, can you teach creative writing in an adaptive learning course? In some ways it’s equally suited to it because I can imagine you could build a pretty good workshop in create a writing course, with a lot of work, but you can do it.

John: I looked at RealizeIt’s website and it said they create unique formative assessment items based on instructor provided question templates. How does that work?

Chuck: It works very well. In terms of doing it, I’ll tell you one of the things, yeah Ryan Baker wrote a really good paper. The technology is developed really well but the assessment in general is still kind of heuristic, if you know what I mean by that, we’re gonna assess your competency by whether you get four or five items right. That’s the heuristic part of that. You have to design better assessment devices. What we are doing now is we have to transform the assessment paradigm in terms of what they look like, in terms of are they authentic, are they reflective, and are they contextually relevant? Students respond much better to questions that are related to the disciplines that they’re going through, and we have to develop that. We’re nowhere near that, but we’re working on that. They’re very good about helping them. You give them a template, we want to do this, we want to do this in a simulation platform, they’ll help you work with it. They’re very good about doing that, but they can’t do it all. It is a partnership.

Rebecca: We usually hear about adaptive learning in online context. Have you had any experience in a hybrid environment or an in-person environment?

Chuck: Oh absolutely. We’ve taught some blended, adaptive learning courses, and it makes absolute sense. What’s a blended course? What do faculty members, the first time they think about blended, and they think about it incorrectly. What am I face-to-face, can I offload to the online environment? That’s about the worst way to go teaching a blended course, right? The thing is you look at what are the appropriate kinds of things for these two different formats? We have this all of the time where essentially we’ve had a blended nursing courses where they do material content offline and come in to the class and essentially do problem solving. It’s basically is very, very appropriate for this notion of a flipped blended course. We’re actually having a faculty member do it in statics, in engineering, which is really exciting. You know, it’s a whole notion of some of the courses are flipped some of the courses are flopped. [LAUGHTER]. But we sort of look at that and it makes kind of sense. But again, back to your question, is where is this modality most appropriate? Where does it fit? That’s the kind of question, and that’s really a departmental discipline sort of thing. Yeah, every discipline believes they are unique, right? Yeah, their pedagogical issues are unique to it. Yeah right [LAUGHTER].

John: Actually, along those lines we’ve had a number of reading groups here and we’ve had faculty from different departments get together and the faculty who are new to this type of thing expressed surprise at how common the concerns that they had were. To find out that people in other disciplines, very different disciplines, faced exactly the same problems and sometimes had some really good solutions.

Chuck: Yeah, we do this thing, I’m very fond of a concept by the sociologist, Susan Lee Star, she called it a boundary object. A boundary object is something like this, we do it all the time at UCF; critical thinking. We’re really into active learning now at the moment and active learning is another boundary object. We’ll do this say with the Faculty Senate, who’s in favor of critical thinking? Of course every hand goes up [LAUGHTER]. You wouldn’t dare say you’re not in favor of critical thinking. But when you get in a large community of practice nobody agrees what it is, right?

John: Exactly.

Rebecca: Yeah.

Chuck: Exactly. So, Susan Lee Star, this boundary object, is something like that. That it holds a community of practice together, but it’s very weak. It’s not strong enough to be really functional in a large community of practice but you go back to individual constituency, go back to physics, or rhetoric, or creative writing, or education, and they damn well know what critical thinking is in their discipline, right?

John: Right.

Chuck: They do. They absolutely can do it and they’re very powerful. When you bring it back to the community you’re back into the same dog fight and that’s a very powerful concept. I can name literally dozens of them: active learning, critical thinking, online learning, you go on. Very, very powerful.

Rebecca: I find writing is one that bubbles up. What do we mean by writing? What does that look like?

Chuck: Oh yeah. What is writing look like when you’re tweeting— are you writing?

Rebecca: Yeah, exactly.

Chuck: When you’re blogging, are you writing? How do you workshop writing now? Are workshops necessary? It’s all fascinating to me, but I guess that’s because we’re academics. If it weren’t for boundary objects we wouldn’t have anything to do. [LAUGHTER].

Rebecca: You start talking a little bit earlier about the implications of adaptive learning at a university. Can you expand upon that a little bit? [LAUGHTER]

Chuck: Sure. I mean the implications is; what does this say for the structure of the university? What does this say for the way when we’ve organized this enterprise called learning? There’s another great boundary object: student learning, student learning outcome, one of my favorites [LAUGHTER]. Go ahead, define that. My friend, Anders Norberg, from Sweden, we’ve written a paper called, a Time Based Model of Blended Learning. Where time becomes a fundamental design structure of a university and it’s virtually very very different from the way we organize learning at the moment and how we have organized learning is in the sense of discrete units called classes, called units, called semesters, called years and called matriculation period time. It has tremendous implications for that kind of structure. We are employed by all of these kinds of things and we’re organized by all of that so sooner or later we’re going to have to re-examine all of that, if we’re going to adopt these things and adapt these things and be environmental like that. There are three things I think associated with good ideas. One is was the adjacent possible, what’s the next reasonable step, it’s what’s next, what can we reasonably accomplish next, that’s the adjacent possible. Outside of that is the adjacent impossible, you can’t do it, you can’t do it. Secondly, you have to have the slow hunch, you have to stay the course. How many things have we done this but we’re gonna try it, it didn’t work right away and then we dumped it? I’m sure nobody at SUNY Oswego has done that, but we certainly have done that at UCF. You got to stay the course! If you read Darwin’s autobiography, he said “Yet a Eureka moment about natural selection when true”. If you carefully looked at his notes he founded months and years for it was in his notes he just didn’t know it was no Eureka moment, that’s stay the course. If you know this, you got to stay the course because it’ll be bad times, everything’s not gonna work, it’s just not going to be the way. Again, you over expect short term, you under expect long term. Then the third thing is you gotta have this liquid network, you’ve got to work with the vendors, you got to work with faculty, you got to work with administrators. SUNY has to work with UCF that has to work with CTU, that has to work with Carnegie Mellon, that has to work with Oleum Geo Services. You have to have this liquid network where we can share ideas. We can’t do it alone, frankly you can’t do it alone, we’ve got to work that way and if we work that way it’s going to change the whole way we do with the business. I guess the question for us is how much do we want to change? It’s way above my pay grade to change a University and I expect for you too, but sooner or later we’re gonna have to accomplish that there’s a lot of implications for the University.

Rebecca: There’s a lot that I’m hearing you talk about, that reminds me of agile design practice. That’s made a big boom in technology and design. The idea of small little sprints break a big problem into smaller problems that you can work towards. And then it’s iterative, you keep going back and it’s circular it’s not a straight line, that’s what I’m hearing you say. That’s the way that it needs to be tackled.

Chuck: I think that’s a good metaphor, we really be in an agile scrum and keep doing it until we get it right. That’s frankly not been our history right, part of our history is let’s declare a victory and move on to something else. I mean that in the kindest kind of way but we have some serious issues in my judgment. We have tremendous educational inequality in this country, unless we crack that, we’ve got some issues that we’re never going to solve.

John: And you guys are doing quite a bit on all those fronts and I hope we’ll have you back for some future podcasts to talk about some of those things.

Chuck: We’ve made some amazing breakthroughs in communities that you think wouldn’t achieve. The talent pool is as deep as anywhere else, but we’ve got to figure out how to let it up.

Rebecca: I think the other thing that I was hearing and what you’re talking about is the idea of these micro credentials is also surfacing that we had a prior podcast on. I really can see the adaptive learning move at the same pace as micro-credentialing because I think they’re directly related to one another and how the system might have to shift.

Chuck: Kahneman wrote that great book “Thinking Fast and Slow” and kind of admitted in terms of one of our habits is we attempt to solve a hard problem; we try to measure outcomes, we can’t figure out what to measure so we measure something else that we know how to measure. I’ll give you a great example “student learning outcomes”. We’re not too good at that, so we measure grades we measure course success, course success is not learning outcomes, we do an easier thing because we don’t know how to do the hard thing. I do that all the time! [LAUGHTER]

John: We all do that… I think.

Chuck: Yeah, I really love Oswego.

John: We enjoy it too, I’ve been here since 1983.

Rebecca: I came back.

John: That’s right! Rebecca was a student as well.

Chuck: Rebecca we knew you’d come crawling back…[LAUGHTER]… Where does the time go John? What happened?

John: I know, I just got here it feels like.

Chuck: Remember John Lennon? That song “My Boy My Beautiful Boy” that’s what happens when you’re busy making other plans.

John: Yes.

Rebecca: All the talk of time leads to the question of, Mike what does the future hold?

John: What are you doing next?

Chuck: What are we doing next? We’re reinventing the University, that’s what we’re doing. We have a new president, president designate Dale Whitaker who’s been our Provost. Dale is a very big thinker and he’s developing ideas like zero probation, all services, the hub for faculty, the faculty center and all services faculty be located in one kind of place that students become an active part of the instructional process. They’re no longer receptacles, if they become teachers as well we have to do that. They have a lot to teach us, it’s a different kind of world to do this. We have to understand better how our students acquire knowledge, we have to better understand the many generations that are existing on our campuses. I kind of like to get my news from “The Onion”.

Rebecca: hahaha !

John: That’s the best place these days and it’s not that far off.

Chuck: My favorite tagline in the onion is eccentric student reads entire book, we have to begin to accommodate the way students learn and use their learning devices. My graduate students at times will send me off to get a cup of coffee when they fix my technology problem. Technology for them is what Machan of the Reedy called “Living now in the InfoSphere” where information communication technologies talk to each other where we’re no longer in the loop, we’re on the loop. And I think you can understand that in terms of when you look at your Facebook page and what you were looking at on Amazon pops up on your Facebook page, you know that these technologies are talking to each other.

Rebecca: I thought they were reading my mind [Laughter]

Chuck: But the cover of last week’s Economist was “Epic Fail” and it was the Facebook F falling off the fail, lying on its back. The covered this week is AI spy, artificial intelligence spying you on the workplace, we have lots going on. I think the recent things that have happened over the weeks with Facebook give us the fact that we have some serious examination of our culture and our information to confront in the decades to come. I think the future is being defined for us by forces outside of our realm. This is pretty scary stuff for me. Thank you both, you guys are great. I really enjoyed it, thanks for having me

John: We’ve enjoyed this tremendously.

Rebecca: Yeah, it’s really interesting to hear what you’re doing.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]