445. Emotional Prosody and Online Learning

The use of instructor-narrated video slideshows has increased in response to both the growth of online instruction and increased use of flipped classroom teaching approaches. In this episode, Corinne Syrnyk and Alyson Kubat join us to discuss their study examining the impact on student learning of audio vs video narration and of the emotional tone conveyed by the instructor’s voice.

Show Notes

Transcript

John: The use of instructor-narrated video slideshows has increased in response to both the growth of online instruction and increased use of flipped classroom teaching approaches. In this episode, we explore the impact on student learning of audio vs video narration and of the emotional tone conveyed by the instructor’s voice.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Corinne Syrnyk and Alyson Kubat. Corinne is the chair of the psychology department and director of research at St. Mary’s University in Calgary, Alberta. She is a developmental psychologist whose research focuses on social, emotional and mental health issues in children, language development, and human-animal activities. Corinne recently established the wellness opportunities and outcomes for pets and humans lab at St. Mary’s. Alyson graduated from St Mary’s last year with a BA honors degree in psychology. Kristin Croyle, the Dean of the College of Liberal Arts, Sciences, and Engineering here at SUNY Oswego, and a frequent guest on the podcast, will be filling in for Rebecca today, who has been called away for some other tasks by other administrators. Welcome Corinne and Alyson.

Alyson: Thank you.

Corinne: Hi. Thank you. Nice to be here. Thank you so much for having us. It’s very exciting.

John: Thank you for joining us.

Kristin: Welcome and today’s teas are… We always start by asking what our guests are drinking today. So are you drinking any tea today?

Corinne: I am not, but I think I will give you the stereotypical Canadian response and say that I started off with the Tim’s, which maybe is too idiosyncratic for your audience, but I went to Tim Horton’s and I got a coffee. But in terms of my preferred tea, that would be fennel.

Kristin: Ooh, nice. And Alyson?

Alyson: I have an Earl Grey.

Kristin: Ah, lovely. And today I’m doing iced tea straight up Lipton with raspberry, since we’re filming after lunch.

John: And I have an Irish Breakfast tea today. We’ve invited you here today to discuss an article that you had published in the November-December issue of Applied Cognitive Psychology entitled “The impact of emotional prosody on online learning in undergraduates.” To help frame this discussion, could you first explain what is meant by emotional prosody?

Corinne: So I think to kind of give you some context for emotional prosody, I’ll start by telling you a story about what we call prosodic speech, otherwise known as “Motherese.” That is essentially the way that we speak to infants. So, when you have a newborn baby, what you’ll tend to find is that the vast majority of parents, new parents, will speak to their kid in a very kind of exaggerated way. So, you have this newborn baby, and you don’t speak to them in a flat affect. You speak to them in an enthusiastic, lilted kind of affect where there’s lots of very clear starts and stops. So you might speak to a baby like, “You are so cute!” You’re relaying a lot of not just words they don’t understand, but the important thing that you’re relating there is potentially emotion and connection, because you’re having that kind of very positive emotional communication with the lilt of your speech, with the tone and the expression. So you fast forward away from prosodic speech to emotional prosody, essentially, that’s kind of the way that we speak, like I’m speaking right now. It has a cadence to it. There is a change in pitch and tone. Hopefully, you can kind of discern my motivation or enthusiasm for the topic by the emotion that I’m conveying through my speech. That is what we would call emotional prosody, and certainly a positive emotional prosody, as opposed to more neutral emotional prosody, which would be a lot more kind of like a flat affect. So if I was talking to you in a monotone way like this, which is a way that you would hopefully not speak to a person all the time, it’s that difference in communication that you’re receiving in terms of the emotional context that’s being communicated. Alyson, maybe you want to build on that?

Alyson: Yeah, I think it’s mainly just focusing on those acoustic cues that accompany speech. So a lot of tone, pitch, loudness of the voice, so all that kind of variance that accompanies speech and can draw interest in people who are listening to the person speak, and just kind of how that engages with the person that you’re speaking to.

Kristin: And what prompted your interest in applying emotional prosody to the online learning environment?

Corinne: Well, I think I’ll just clarify that Alyson was my honors student last year, and she did a really wonderful project. She got into her Honors Program, a competitive program, and I’m very picky [LAUGHTER] with my honors students, and I was very happy to work with her and right off the bat, Alyson communicated to me her interest in this topic, and she’ll tell you more about her own kind of personal ambitions and how this has informed her desire to work with me as a developmental psychologist. So Alyson, tell them your little story.

Alyson: Yeah, of course. I have a very personal interest in speech. I think it’s so interesting. I love learning about language development. I took a class during the… I believe it was the third year of my undergrad… focusing on language development, and the instructor was a speech pathologist. And I absolutely loved that class. It was so interesting to me. And when I went into honors, I just really wanted to focus on that interest in speech and language development. And I also am wanting to be a speech pathologist, hopefully. So I felt that that would be a good topic to focus on. I also, just through my learning, I found that most of my education was done through online environments, I suppose. I graduated high school in 2021, so it was during the height of COVID. So I found it really interesting just how quickly everything just turned into online learning just within the span of about a couple of weeks, maybe two weeks. And so I just found that so interesting, just based on the fact that there was quite a lack of understanding just about the foundations of online learning, about how it will impact learning in students, regardless of age. And I just wanted to kind of expand on that, just to be able to study speech in terms of online learning as well.

Kristin: Thank you.

John: And you began the discussion with the summary of some of the earlier research to set the stage for what the gaps are that your research fills in. Could you give us a bit of an overview of what studies have found before about emotional context and prosody in learning?

Alyson: So I think mainly previous research has focused on, as Dr. Syrnyk mentioned before, infant- directed speech, and just how crucial it is to the development of language in children, and just using that exaggerated speech, I find that’s where most of the previous research was kind of rooted in, considering prosody. And just prosody really aids cognitive processes such as attention, recall, and also promoting concentration. So most of that previous research just demonstrated that it possibly has implications for learning. So a study that I found really interesting when I was looking at previous research on this topic was a study conducted in 2022 by Dylman and Champoux-Larsson. They specifically focused on the impact of emotional prosody on the listening comprehension in, I believe it was 11- to 13-year old children. They found directly that positive emotional prosody promoted greater listening comprehension than when spoken to through neutral emotional prosody. I found there that there was just a very apparent lack of research on older students considering online communication. Yeah, so just the importance of emotions as well while communicating. Something that I found really interesting was considering academic emotions as well, and just how learners understand emotions in regards to their learning. A lot of the time, students felt that the emotions that they were experiencing while they were learning directly impacted their learning performance, as well as their social presence in online learning environments as well. They found, that in regards to social presence, when students perceived that there was a greater social presence in terms of online learning, that students felt that they would learn better, but they didn’t necessarily learn better. So I found all that to be quite interesting in terms of just building up the research for this study.

Kristin: Can you expand just a little bit? What do you mean by social presence?

Alyson: Oh, yeah. So in online learning environments, there was kind of an understanding that the facial cues and then the visual cues provided in online learning, where students were able to see the instructor, that would promote greater communication, and they would feel more connected with their instructor. But that definitely was not the case a lot of the times. So students felt that they were able to connect better with the content that was being spoken to them when they’re able to see a visual of the person speaking to them, but a lot of the times that actually interfered with their learning just based on the stimuli that was interfering with that.

Kristin: Mmhmm, kind of the number of things that they had to pay attention to.

Alyson: Yeah, it definitely impaired more than helped, even though a lot of students personally felt that that increased social presence would benefit, whereas it actually interfered with their learning quite a bit.

Corinne: Yeah, so like the social presence, this notion that somehow the instructor being visible is helpful is just kind of a wide held assumption. It really hasn’t been tested very much, which is part of the reason why Alyson wants to look into this as part of her research here. But there is a lot of mixed evidence in terms of the limited studies that exist in this field of interests. There is mixed evidence as to the efficacy of social presence. We don’t really have a hard yes or no as to whether it’s more beneficial or not, particularly when it’s combined with emotional prosody. So one of the kind of really neat things about her study is that she of course, got to kind of collapse these variables together by seeing what the impact of emotional prosody, whether positive or neutral, would be in impacting recall in terms of student performance, when you look at whether that social presence is activated or not. I also just want to take the opportunity to expand a little bit more on that Dylman, Champoux-Larsson 2002 study. That was Alyson’s kind of guiding light in terms of setting the foundation for her research. She came to me with that study very early and said, “I want to do something with this,” [LAUGHTER] and we really worked from that study quite a bit. Just to kind of build on that a little bit more. It’s really interesting that there’s a very limited scope of research when it comes to emotional prosody, and Dylman, Champoux-Larsson’s study is the one to go to. And what they found in that study was that they looked at younger kids, a group of 8- to 10-year olds versus slightly older, and the older were 11- to 13-years old, and what they did is they essentially tested them on five open-ended questions that were related to their content learning in their classes, which was grade appropriate. And from that, what they effectively found is that emotional prosody was important for the older kids, the 11-to 13-year olds, but not so important for the younger kids. When you think about the developmental profile of coming from a place of prosodic speech being important to kind of maybe having a little bit of a U-shape across middle childhood into later childhood.

John: I think the study is especially interesting given how much more important online learning has become, as well as even in classroom learning, more and more instructors are using videos and using flipped learning approaches. So this applies to a growing share of the instruction that’s taking place. Could you describe the design of your experiment, the four conditions that you used?

Alyson: Yeah, absolutely. So in terms of the four conditions, we were comparing positive emotional prosody versus neutral emotional prosody, and then we were also comparing the delivery conditions, the delivery lecture conditions, so it was a video condition and a audio condition. So based on that, we looked at the interaction between all of those conditions as well. So all participants were shown a 11-slide PowerPoint, and then within that, they were either presented with a video version or an audio version. So the video version would have a video of the instructor teaching on the upper right-hand corner, whereas the audio condition, they were just presented visually with the lecture slides.

Kristin: Nice and tell us a little bit about your participants.

Alyson: So the participants were 80 undergraduate students at St. Mary’s University, and within that there were 20 students assigned to each condition group. They had a mean age of roughly 20 years old, and they spanned all of the degree programs within the school.

Kristin: And I know somebody is going to want to know what the lecture was about. So tell us what the lecture was about, the 11 slides.

Alyson: Yeah. [LAUGHTER] So the lecture… pretty much when I was looking for a topic for something to teach, I wanted it to be something that could be taught relatively quickly, within the five- minute time period that I was looking for for the lecture. And I also wanted it to be something that was slightly more of a niche topic, which proved to be a bit more difficult, just in terms of finding a topic that I felt would work well under those kind of guidelines that I had set out. So I eventually turned to astronomy, and I looked at astronomical eclipses because I felt that was slightly more niche that the majority of students wouldn’t have been previously taught that topic, whereas, while I could still kind of maintain that five-minute duration while being able to include all the information that I wanted to quiz them on.

John: So what did you find when you compared the outcomes in terms of how much the students remembered from these videos?

Alyson: So we assessed them based on a 10-question multiple choice quiz. So once they watched the lecture video, they would complete that quiz directly after. And we found that in terms of the results of the study, that the participants who had been given the positive emotional prosody condition performed significantly better on the quiz than the students who had the neutral emotional prosody so even just comparing that, the positive emotional prosody condition had a mean score of 72.25% on the quiz, whereas the neutral emotional prosody conditions had a mean score of 56.5%. And we found there was really no significant difference when just considering the delivery of the lectures, comparing audio versus video. But the interaction was quite interesting that we ended up finding. So I had initially assumed that combining positive emotional prosody with the video lecture, just due to that potential increased social presence, would result in a better score on the quiz. It actually turned out that while emotional prosody, when conveyed positively increased the quiz scores, it was actually that audio condition that resulted in increased grade scores compared to all the other conditions, which I thought was quite interesting.

Kristin: So if there was positive emotional prosody, if it was the positive emotional prosody condition, they had significantly higher scores on the quiz, as long as there was no video, yes? Did I get that right?

Alyson: That is right. Yes. [LAUGHTER]

Kristin: It sounds like it was a surprise. I know that you have some explanations about why that might have been the case. What do you consider to be the most promising explanations?

Alyson: I think overall, I found the cognitive load theory to be quite helpful with explaining that interaction, and it was really just focusing on the amount of information that was presented to the students and how that impacted them cognitively, in terms of what they were able to retain. So I think for most of the students, it just kind of helped that the audio condition was in combination with positive emotional porosity was interesting to them, enough to engage them cognitively, but it wasn’t at the point where it was exceeding their cognitive threshold, where it would be negatively impacting their cognitive processes such as attention and recall and comprehension, whereas I think it was the video condition that really was just too much in terms of cognitive threshold, and it just exceeded that threshold and eventually just negatively impacted their grades.

Kristin: So kind of a sweet spot of attention without too much, without going over in terms of what they were being asked to pay attention to, both the facial expressions of the instructor, for example, and the content.

Alyson: Absolutely, yeah.

John: One way of stating this is that the video of the instructor was distracting them from the content on the slides?

Corinne: Yes, exactly. And I think I’ll just give you a bit of a definition for cognitive load theory. It’s essentially where we get a limited amount of information that we can process and that varies from individual to individual, of course. T here’s a normal distribution that most of us can maintain attention, maintain active learning with a certain amount of information, some of us more so or less, but we each have these individual thresholds that are reached, and when that threshold is reached, that’s essentially your working memory bench or table is full, and you can’t really process much more. So your ability to attend and comprehend learning what’s going on, it’s kind of maxed out. So yes, you’re quite right, John, like this notion that there’s this point by having that additional information of the instructor visible on the slides was just too much, and one way to conceptualize that is through this notion of distraction. When you have too much information, it is distracting and you can’t absorb the information that you probably could if there was less information there.

Kristin: I know that your study doesn’t go this far. I expect that people who are teaching online, who are using a lot of video, that they would argue that there are other benefits that they’re getting from that, that they’re, for example, trying to form an individual relationship across space and time, and that they’re using it instead as a motivational tool, or something like that. I know this is beyond what your study looked at, but what are your thoughts on that?

Corinne: I think that’s a great point. This is part of what Alyson and I were interested in. There’s this kind of general… ever since COVID, particularly… this general assumption that being visible is better, like as an instructor, but we have very little evidence that it’s real. In fact, this study kind of shows the opposite pattern of effect. So, in terms of adding the video, the video yourself, as John says, might be a distraction. This study is a small study. Of course, it needs replication. We need larger sample sizes, of course. But what I tend to see in terms of the online learning world, I’d like to see more evidence-based pedagogy, particularly when it comes to things that are related to like attention and memory and recall, because those are things that we naturally depend on, we take for granted in a classroom environment, because you can see your students and you can engage with your students, and you can tell if somebody is on their phone or asleep. But in an online environment, of course, that feedback is often one way, and the assumption is that, well, if they see me while I have these slides on, this is going to be better. But do they really need to see you? Maybe not. Do they really need to hear you speak with some enthusiasm or emotion when you’re speaking? These results seem to support that instead.

Kristin: I’m struck, because right now, in this environment, the listeners for the podcast are going to be listening to people with positive, emotional prosody and no video. But right now, as we’re recording, we’re using a platform in which we can see each other on screen. So we’re doing our best to replicate your conditions [LAUGHTER] right here at this moment. We’re trying to leave out the negative, the lack of emotional prosody though.

Corinne: There’s so much scope for more research here, really, which is really exciting about Alyson’s work here, is that it’s really just scratching the surface of what else we could really be examining within this area.

Kristin: The other thing that it brings to mind for me is research on distracted driving. That conversation in the car is less distracting than a phone conversation, for example, partly because the other passengers recognize the driving conditions, they can see if you’re entering a difficult driving situation. That perhaps it speaks more to the actual reality of being in the same place and time in which that personal connection makes a bigger difference. Perhaps there is something different about the online environment where we’re trying to create an individual connection, but at the same time, there is an artificiality to it, and prioritizing student learning over our assumptions about what might be creating a connection may be a more supported way to go.

John: I was really impressed by the magnitude of the effect, not just that it was statistically significant, but that was a really large difference in the scores. That result is remarkably strong.

Alyson: I was quite surprised by it as well. I think I was expecting there obviously to be some difference, but when I calculated all the quiz scores, I was genuinely surprised by how significant of a difference it was, especially when just considering the fact that it was a five-minute lecture video and immediately followed by a 10-question multiple choice quiz. I really wasn’t expecting the results to be that significant, especially just considering, yeah, like you said, the lack of research, the lack of basis that there was in terms of providing significant support, I suppose. Whereas it was just kind of surprising how significant that difference was.

Kristin: I’m guessing you triple, triple checked it. You’re like, “what?”

Alyson: I did, yes. [LAUGHTER]

Kristin: Two full questions better, yeah.

Alyson: Yeah, exactly. It was just very, very surprising. But it was really interesting to go from watching these people engage with the study and then have actual results to follow that I found that to be quite interesting.

Corinne: Alyson did a really strong study, I’m really proud of her. She had a really strong design, and she was really committed to recruitment and seeing through all the very important aspects that are required when you’re conducting research. And she did such a fine job that we are so happy to be able to go ahead and publish her work, and work with people like you as well, so that we can share these results.

John: And we should thank Michelle Miller, who wrote this up in her substack earlier, which is how we found out about this paper. We’ll include a link to your study and the studies you cited, as well as to Michelle Miller’s substack post on this, for people who may want to follow up a bit more. Going back to Kristin’s comment about how our podcast is audio, I did hear a statistic, actually, from some of our colleagues at Finger Lakes Community College this week, where they noted that 40% of podcasts have moved to a video format. We have chosen not to do that for a number of reasons. The main one is we mess up a lot, [LAUGHTER] and this way we can edit out all the mistakes before we release it, but also, we like to listen to podcasts with just the audio, and both Rebecca and I at least, generally don’t watch videos quite as much because it takes more time to sit there and watch the video, when with a podcast, you could be driving, you could be walking, you could be doing other tasks. For us, we’re probably going to stick with it, even though we do see more and more podcasts moving to a video format. But at least now we have a little bit of empirical evidence to support the continued use of audio only.

Kristin: And we always end by asking: “What’s next?”

Corinne: Well, like I said earlier, I think that Alyson’s study here really just opens up a whole can of worms for all sorts of very exciting new leads into research. One of the things that Alyson and I certainly have been discussing is we’re assessing this in terms of student recall or performance on a task, but it would be really interesting to look at how students perceive themselves, emotional prosody, within online learning mechanisms. Because I think it’s something that I would assume that most students pick up on and have preferences based on the emotional prosody of their instructors, and that would be reflected in terms of whether they think that instructor is a better quality instructor or not. And what does that mean in terms of the content that they actually relay? And this is another thing that we were thinking about, which was really maybe a little bit off the wall, but kind of interesting. You’re familiar with the uncanny valley phenomenon, right? And we’re thinking, what if there’s like, an uncanny valley for emotional prosody? What if trained people who are more neutral on their emotional prosody. How would that be perceived by learners or by people listening? Would they be convinced necessarily, or would they be like, “Oh no, there’s something going on here?”

Kristin: For listeners who may not be familiar with the uncanny valley. Can you quickly describe it?

Corinne: Yeah. So the uncanny valley phenomena is this notion that humans are essentially kind of very suspect [LAUGHTER] to robots. We have a high sensitivity to the artificial. We’re very kind of good at sussing out what’s artificial versus what is not. And that is found across a range of kind of things in the world. So whether it’s robots or different kinds of aspects of learning modalities, we’re good at picking up on artificial versus real. And this is an effect that’s evident in infancy, like it’s evident from very early on. Very young children will be able to discern inanimate versus animate objects, inanimate versus animate movement, etc, physical properties. And this extends into, of course, you know, as you get older, but in terms of an uncanny value, and when it comes to emotional prosody, and not all instructors are built the same. You know, I think about all your university profs. I’m sure we all had University profs, I’m one of them, that had a more kind of flat effect when they talked, they spoke more monotone and they kind of talked like this. They probably had a reputation for speaking a little bit more monotone, but in today’s world in higher education, there’s certainly a lot more emphasis on pedagogy, on teaching and improving skills and attributes of our instructors, which is great. But can you teach emotional prosody? Can you teach someone to speak with more emphasis and cadence and expression? And if you could, what impact would that have on people?

Kristin: Or could you essentially auto tune them into it in an online environment? Would that be perceptible? How would it be perceived by students if they knew or didn’t know? I mean, we see what auto tune does to music, it’s used sometimes in the background and is an embarrassment, if people find out, other times in the foreground, as a musical technique. With all the tools we’ve got, certainly there would be a way to manipulate it.

Corinne: Absolutely. Fascinating, you can do all sorts of things, and with AI now too, I mean, right? Like we are in this weird world of we’re not quite sure what’s real anymore or what is and I don’t know exactly what that would do to things. You could probably try to set up different aspects of non- real people doing this kind of thing now, and test that in terms of perception and learners or people in general.

John: And ElevenLabs and some of the other AI tools have developed some very realistic, emotional prosody in terms of their speech. And it would be interesting to do an experiment when people are faced with AI avatars delivering that versus perhaps actual instructors. [LAUGHTER] Because we do see AI avatars being used increasingly in a lot of social media, a lot of ads and so forth, and I wouldn’t be surprised if they started showing up in some online instruction.

Corinne:Yeah, kind of terrifying, [LAUGHTER] for a number of reasons. But yeah, there’s a lot of questions there. And I mean another thing, I love research, so when I started thinking about all the other things, you know, it gets really exciting. But like, at what point does emotional prosody compete with content? Like, there has to be some type of like, I would assume there has to be some type of U-shape there in terms of that effect. I kinda think about, like, people who are very convincing speakers in terms of the way that they speak with that emotional prosody, but the content is incongruent with the actual delivery. How is that interpreted? What impact does that have on recall of the message that they’re giving? So yeah, lots of really cool things to explore.

John: And I do have to say, when I was reading this, one of the things I was thinking about was the movie Ferris Bueller’s Day Off, and that little video clip, that little segment in the middle where Ben Stein was talking about tariffs and how harmful tariffs could be in the economy. And apparently that message never got through to students, or we wouldn’t be in the current situation we are in the U.S. Maybe it’s because we tend not to have enough emotional prosody in our economics classes.

Corinne: Well, I want to say anything, but, [LAUGHTER] yeah, there’s some weird stuff going on there. [LAUGHTER]

Kristin: That is really funny. [LAUGHTER]

John: And so Alyson, what are you doing now?

Alyson: I am currently working as a 911 dispatcher with EMS. I am hoping to apply to graduate school in the next year or two to kind of pursue that. I am hoping to volunteer as well within the next bit, and also just hopefully to engage in some future research opportunities. I think just working with Dr Syrnyk through this whole experience has been very helpful in terms of mushing together my education, I suppose, in terms of all the courses I’ve taken in the past about research. It’s really shown me how important research is, and especially the discussions that can result from research that occurs. So I think that it’s just really made me quite passionate for future research opportunities that I could possibly have, where the research that we’ve done here could possibly lead. I think that would be really interesting to see.

John: And you’re off to a great start already doing some research that can have a significant impact potentially on how people teach online classes and how students learn. Anything that we can do to help improve that would be really useful.

Alyson: Yeah, absolutely. Especially just as online learning becomes increasingly prevalent in university classrooms. It’ll especially be interesting to see how different forms of lectures are incorporated into online platforms. There are some classes that I’ve taken, like a statistics course, where it’s very much more involved. It’ll be interesting to see how that kind of forms from where it is now into just based on research, distant into the future.

John: Well, thank you. We really enjoyed talking to you, and thank you for sharing your research and this discussion of your work with us.

Kristin: Yes. Thank you both.

Corinne: Yes. Thank you for having us. It’s been wonderful and nice to get to know you a bit and see you. It’s just been a great opportunity.

Alyson: Yeah, thank you so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing Assistance provided by Madison Lee.

[MUSIC]