Casey Raymond, the 2017 winner of the President’s Award for Teaching Excellence at the State University of New York at Oswego, joins us again in this episode to discuss how he uses Excel spreadsheets to generate algorithmic questions in Blackboard Learn.

### Show Notes

- Brown, Peter C.; Henry L. Roediger III, and Mark A. McDaniel (2014), Make it Stick: The Science of Successful Learning. Cambridge: Harvard University Press.
- Lang, James M. (2016).Small Teaching, Everyday Lessons from the Science of Learning. Hoboken, NJ: Jossey-Bass.
- Miller, Michelle (2015). Minds Online: Teaching Effectively with Technology.Cambridge: Harvard University Press.
- Raymond, Casey (2017). Casey Raymond’s January 12, 2017 workshop at SUNY-Oswego on Creating numerical questions in Blackboard Learn.

John: There is an overwhelming body of evidence that supports the use of retrieval practice to encourage long-term learning. While this has been known for decades, this was once very cumbersome and time-consuming to implement. In this episode, we’ll explore how learning management systems allow students to have unlimited practice.

Thanks for joining us for Tea for Teaching, an informal discussion of innovative practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist

John: and Rebecca Mushtare, aa graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

John: Today’s guest is Casey Raymond, the acting director of the Honors Program, an associate professor of chemistry and geochemistry, and a recent recipient of the President’s Award for teaching excellence at the State University of New York at Oswego.

John: Welcome, Casey.

Casey: Thank you John and Rebecca.

John: Our teas today are:

Rebecca: Jasmine green tea

Casey: elderflower Rooibus,

John: and Twinings English afternoon.

Rebecca: So, Casey, we know that you use low-stakes testing quite a bit in your introductory level courses. Can you talk a little bit about why low-stakes testing is so important for these particular courses?

Casey: So, particularly with general chemistry, the only way to get good at it is by practice and just telling the students to do questions at the end of the textbook doesn’t really work because they don’t do it. I could collect them all but I don’t want to grade all of those. I could grade just some, but again it’s still me grading them and so being able to do something electronically that encourages… forces… students to do something… is a really good avenue to get them to practice.

John: I don’t think that’s anything unique to just chemistry. There’s a tremendous amount of research out there involving the importance of retrieval practice in effective learning in pretty much all disciplines. Here at Oswego, we’ve had a series of three reading groups where that was a common theme to each of those. We had a reading group on Michelle Miller’s Minds Online: Teaching Effectively with Technology, and then we had a second reading group on Make it Stick, and then our current one is on Small Teaching: Everyday Lessons from the Science of Learning. Each of those has a significant portion on the learning gains associated with retrieval practice.

Casey: Yeah, to a degree, these are the modern version of flashcards kind of thing. It’s just a way to really get the students doing the work… and I announce in class if you’ve had to do a question 20 times and still don’t get it right or 20 times just to get it right, you probably need to be doing more questions like that and then go to the book, go to other textbooks, go to resources the publisher has available but it’ll help you determine what you don’t need to study or what you do need to study more.

Rebecca: Yeah, I think students often have no idea what\ they know or don’t know.

Casey: Yeah, and when it gets to the test and they realize that, it’s sometimes, it’s too late.

Rebecca: Yeah, that’s not not a good time to figure that out, right?

Casey: No

John: How do you weight these? And how many chances do students have to work on these problems?

Casey: So when I do this for homework, most of the time I give them unlimited attempts but there’s enough randomness in there that it’s unlikely they’ll see the same questions again… and my homework are usually two, three, or maybe four problems is all… but I’ll offer on average probably three homework assignments every two weeks, so I’m doing a few questions very frequently and in total that’s worth fifteen percent of their course grade. I do weekly quizzes. They’re usually over the weekend and again there are two or three questions is all, and that’s worth fifteen percent of their overall grade… and then at, by comparison, their midterm exam, their one in-class midterm is fifteen percent of their grade. So, in reality, these are all weighted equally. The homeworks, they usually can do as much as they want, I’ll take their highest score. The quizzes, I give them two attempts… and in some cases I do their best score. In some cases I take an average so I tell them if you get it right the first time, don’t do it again and so the quizzes are a little more information in terms of do they really know it because they only have the two attempts… and if they’re doing well on those they should do well in the midterm….

But, there’s always the students that have really high online scores because they’re using every possible resource to get the right answer… and then they get to the midterm and they don’t have all those resources and they score low because everything there is equally weighted. They still have plenty of time to improve before they get to the final exam.

John: …and one nice advantage of allowing unlimited attempts and keeping the highest grade is there’s always an incentive for them to do it again… so if they want to go back and space their practice and review (which is also important in terms of learning. If they want to do that they can go back and redo things from earlier in the course without worrying about the chance of blowing their grade.

Casey: Right. Yep, and what I typically do, though, as well as kind of impose some due dates is I just make a copy of the homework assignment and label it as practice…. and move all of the point values to zero…. and so once the homework two closes, I open homework two practice for zero credit… and it’s categorized differently in the gradebook in Blackboard.

John: OK.

Casey: So it has no bearing whatsoever on how blackboard calculates their grade and it gives them infinite practice after that. I also allow them, on the practice version, to see all of the answers, whereas when the homework is in play, it just lets them know which question they got right and which question they got wrong. They can’t actually see the answer…. how close they were…. and part of that is, in some cases, there’s limited randomness built into the problem. So, you know, I want them to really have to be working through it, but after the fact, when they’re just practicing…. getting ready for the midterm… getting ready for the final…. they should be able to see the answers.

Rebecca: Can you talk a little bit about how you implement the kind of low-stakes testing in our learning management system?

Casey: So, I really got started doing this when we were under ANGEL as our learning management system. I used other learning management systems before coming here, but when we, as a campus, settled on using ANGEL, I realized it was viable again to do some low-stakes testing and worked with Jeff Snyder to develop ways to write algorithmic mathematically based problems where there was really infinite variability in the question…. and we actually got pretty creative and wrote some pretty complicated questions within ANGEL…. and after Blackboard purchased ANGEL and we knew that it was likely our campus would shift to Blackboard, I had a sandbox given to me to start testing and we immediately recognized that most of what we did in ANGEL was going to fail horrendously in Blackboard…. and so it was, what do we do now? And part of my time during sabbatical was working out ways to do some of the things we did in ANGEL within Blackboard…. and I got a SUNY Innovative Instructional technology Grant to put that all together as tutorials… which most of the hard copy is complete now. It’s a matter of going through and implementing the figures and probably some short video tutorials on how I do that… whether it’s multiple choice questions or numeric questions…. but ways to get around the limitations that Blackboard presents us…. which I’ve been able to do in all but one situation.

John: How have students responded?

Casey: Overall, I think they’re okay with having the homework online. There’s certainly a learning curve for the students understanding that it’s a computer doing the grading and that there are limits to how much the computer can think. So if they don’t give the response the same way Blackboard is expecting it, it’s marked wrong and so part of it is educating the students in that context, and part of it is writing the questions such that there’s less ambiguity in entering the answers. I think initially the students don’t recognize the value of the low-stakes testing model, but once they, I think, start getting towards the midterm and see that…hey, those questions I had in homework and quizzes are just like questions on the midterm, I think they start to realize the value of it.

John: I’ve had a similar issue when dealing with algorithmic type questions where I have a question pool with a mix of variants and one of the problems, as you said, is if you have free response questions, students will come up with creative ways of giving you answers that you may not have anticipated. I’ll often have a list of ten or fifteen different possible correct answers and students will find new ways of writing it. Then I have to go back and vary that a little bit or search within a string. Have you been updating the questions or just giving them more guidance in the question itself.

Casey: For the most part I’ve been giving them more guidance in the question and especially the very first few homework assignments tend to be simpler… tend to be more straightforward… and to introduce that “how do you enter your answer?”… and in the sciences it’s really important to us that students include units in numeric responses and we really want to see that on their written work in their answers on it exam, but units are a disaster for online systems because there are so many different ways that you could write the units that I can’t even begin to fathom how many variations or iterations I would have to try and come up with to avoid problems …and so we’ve taken the stance of just not asking for units on the questions we asked the students to give your answer in nanometers or give your answer in meters or kilograms. We tell them what units we want the answer in, but we don’t ask them to put the units label on it.

Rebecca: Are there other workarounds you’ve had to also kind of adapt for this system? Some tips that you might have for others?

Casey: So to deal with the mathematical problems, the limitation that we encountered with Blackboard is that it only allows one dependent variable. I mean there were some limits with the way ANGEL l worked that we dealt with, but the real hurdle is you can only have the one dependent variable when you…

John: this is in terms of algorithmic…

Casey: in terms of algorithmic or even numeric questions… any kind of mathematical problem — and so if you wanted a multiple choice question that had a number as an answer and you wanted the numbers to be different for each student, Blackboard can’t do it or as an example if you had a string of three percentages blackboard could randomly generate two of the percentages but it can’t figure out what the third percentage is if you need to calculate something else in the question because that would be two dependent variables so what I’ve determined and figured out and really have mapped is to use spreadsheets whether it’s Excel, or Numbers, or Google sheets… it really doesn’t matter, but you can code your questions in a spreadsheet and then export that as a text file that can be uploaded as a pool of questions into Blackboard.

Now the nice thing about it is once you’ve done the question once and it’s all on one row of your spreadsheet all you have to do is copy and paste it to as many versions as you want. I’ve done as many as a few hundred versions of the same question this way and so there’s all different sets of numbers that the students could get and then I upload that set of let’s say 250 versions of the question as one pool of questions in Blackboard…. and so when it’s time to write a test, an assignment, a quiz – (Blackboard calls them all tests) — I can go to that pool and say pick one of any of these 250 it’s a good.

Rebecca: So, the time for you mostly is up front getting those questions set up?

Casey: And so most of the time is up front writing them in Excel and if you can write it out on paper first, once you learn a few key functions within Excel and how to handle a few things in Excel, the time really is just copy and paste and getting the text file created and uploaded. One of the other things that it allows you to do with the Excel is use HTML coding within your text so that, if you want a table of values presented to the students, you can do that, and the students will see a table of values within their Blackboard problem. Or you want to use any other kind of formatting or symbols, you just code it all with the HTML and it will upload with the text and Blackboard understands it.

John: When you have distractors on your multiple-choice do you generate them using common misunderstandings or mistakes that students make?

Casey: …and so I rarely use a multiple choice question, but I did figure out how to do it because I know many people will want to know how to do it, and so, yes when you create a mathematical multiple choice question you can have it calculate the distractors as typical mistakes that you know students are likely going to make: a negative sign, a flipped division, an improper conversion between units, all of those can be incorporated and there are ways within the Excel to basically set it up so it does a random representation of the correct and incorrect assignments. So it’s not always version B is the correct answer.

Rebecca: There’s a lot of nuance to kind of the workarounds. Do you have these in a in a format that you can share out?

Casey: Yeah, that’s the documents that I’m kind of creating now, and the steps and I’ve decided and made the choice in doing these to kind of present it at different levels. Kind of… first level is just a set of text of this — is what I’m doing — just kind of prose. If somebody really understands Excel or understands the concepts, they can read the prose and go apply it to a few more key steps that are required for any of this that you know walks through how to do it… and then my goal is with the videos is to actually go through step by step and show people this is how you do one. For those that might not be so comfortable with spreadsheets or using HTML and then the very last part of the document is going to be some of the really common code that you might want to use like how to make a table in HTML or what are some of these special characters and how do you code those, especially ones that you might want to frequently use, and so the goal would be people can just copy and paste that into whatever text they want to write and not have to go find it or really learn how to code HTML.

John: In the show notes we’ll include links to presentations that Casey has done on this as well as a link to these documents once they are available. So we’ll be updating those as things as things are posted.

Casey: Yep, and then you know I’m happy to share the kind of the initial ones that I have so far as a draft.

Rebecca: Great! So our last question always is: what are you gonna do next?

Casey: I don’t know that there’s going to be a next right now with Blackboard. You know I want to get these documents and tutorials and such in place and I don’t foresee what the next is and this one… let me put the bed huh.

John: Very good. Well, thank you, Casey!

Casey: You’re welcome.

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer.