28. Augmented reality

Show Notes

• Tag AR
• R Studio (Renée’s design studio)
• Metaverse (referred to as Meta in the podcast)
• Pokemon Go
• Snapchat
• Swift
• Yelp
• Zombies, Run!

Transcript

John Does reality sometimes fall short of your expectations? Perhaps it’s time to augment your reality. In this episode, we discuss the creation and use of augmented and virtual reality experiences that can increase our productivity, overcome cultural and language barriers, and provide a richer learning environment.

Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca This podcast series is hosted by John Kane, an economist…

John …and Rebecca Mushtare, a graphic designer.

Rebecca Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

Rebecca Our guest today is Renée Stevens, an award-winning Interactive and Motion Designer and Assistant Professor and Associate Chair of Design at the Newhouse School at Syracuse University. In addition to teaching, Renée also runs her own design studio, is an exclusive designer for Minted and the co-director of education for the upstate New York Chapter of AIGA, the Professional Association for Design. Welcome, Renée!

Renée: Thank you.

John Today’s teas are…

Rebecca English afternoon.

Renée: Chai!

John Republic of Tea’s Emperor’s White Tea. So could you tell us a little bit about augmented reality? How does it compare to virtual reality and mixed reality and so forth?

Renée: Sure. The biggest difference between virtual and augmented reality is that virtual reality is a fully immersive experience that actually gives you a completely new view and a full inclusive view of another place. So, you could be fully immersed and you have 100% of your attention focused elsewhere, versus an augmented experience which is basically a layer of information that is applied onto the world around you. So, you are getting additional information, but yet you still have all of the things happening in your environment including your sights and sounds that you can then layer information on top of. And then of course you add that to your mixed reality which is kind of just a glorified augmented reality… where it’s a little more technical and a little bit more computer graphics based… a nice happy marriage between virtual and augmented reality.

Rebecca For those that maybe haven’t had an experience with augmented reality and can’t quite envision what you’re talking about, can you describe an augmented reality experience?

Renée: Well the one most people know would be Pokemon Go for better or worse, but that’s one that most people usually have a connection to. Snapchat also has some augmented experiences, with stickers and filters and things like that. Those are the ones that I think are the most mainstream that people understand. But, essentially, it could be something as simple as just adding navigation into the view where you’re driving… having it look like it’s in the road in front of you… or it could be something like using your mobile device to learn about something new in front of you… like a new device…. like how to turn a coffee machine on or something like that. So, it’lll apply an additional layer of information that makes the task at hand easier.

John And that information could be triggered by visual cues, by your phone’s camera, or by geospatial coordinates.

Renée: Yes, absolutely. So yeah, it depends on the function and obviously the users of the app. But yes, it could be based off of the camera… actually tracking a specific location in your environment… or your actual geolocation…. or visual cues or taps… or interactions with the user.

Rebecca What got you interested in augmented reality?

Renée: I was bored, no. [LAUGHTER] Not really.

Rebecca I’ve seen your agenda. I don’t think you are bored.

Renée: I wish. When I actually like sit down and think about it, it’s really been a perfect kind of combination of all my backgrounds. I started off with my undergraduate in graphic design, so I’ve always had this love for design. I obviously love to teach because I’m a professor, but I’ve also have a master’s degree in photography and specifically multimedia and storytelling. So, in my undergraduate, I was focused more on the foundations and the principles of good design practices and that led me in towards being more of a user experience designer and user interface designer. But my love for story and all that kind of got me into motion design, and so when you combine motion design and the user experience design… Merged together, that’s like the perfect marriage of augmented reality. So, I get to create mobile experiences and that kind of UI/UX experience, but with my knowledge and love of storytelling and designing for 3D space using time and interaction and all that good stuff. So I almost got into it by mistake, because I was just starting to do all these things and had all these ideas and I was trying to find a platform to make them come to reality. It turned out that’s in an augmented one.

Rebecca Nice… Nicely done. [LAUGHTER]…

John Nice segue.

Renée: I never said that before, it just kind of came out… so that was good.

Rebecca One of the barriers, I would imagine, in getting into this field as a designer, is having technology or packages available so that you can actually enter into this field and so the timing seems like it timed when… I think you and I talked about this previously… that it timed well when Apple released their AR kit.

Renée: Yes, so I actually had this concept for an app called “Tag AR” and it wasn’t called that at the time, but I had this idea and I was trying to make it come to fruition and I couldn’t get the technology right to do it. I had this idea, this concept… how is this going to work? And I didn’t know exactly how that was going to happen and I was actually talking to developers before AR kit came out from Apple and they were all like “we don’t have a solution for that, we don’t have a platform to release this.” So, I was kind of waiting for something to come along and that’s when ARKit came out. That day I reached out to all those developers and said “Ok, now we have the platform, who’s ready to do this?” and of course they all looked at me like “we don’t know how to do this” and I’m like “well, no one does, that’s the whole point.” So, yeah, the timing of that was great because I had this idea, this concept, and just needed the technology to back it and it all came within weeks, if not even days, looking for that perfect solution.

Rebecca How funny.

Renée: I know. It was meant to be, it’s kind of like one of those things where you don’t ask questions, you just go with the flow, like clearly that’s the path, so I just took it.

John And you’re doing the programming yourself?

Renée: Yes, so I ended up developing it myself, I couldn’t find a developer like I said. They said “you know, we don’t know how to do that, it just came out” and I said “yeah no one knows how to do it”. So I struggled trying to find the developer, and I’ll actually give my husband the credit, he said “you do what you always do” and I said “what’s that?” and he said “you do it yourself.” So, he may regret that now, seeing how many hours I put into development, but yeah that was like: “okay, I can do that” and he just gave me that extra confidence and I have been developing the whole thing independently since.

Rebecca Can you tell us a little bit about Tag AR?

Renée: Tag AR is an augmented, “hello, my name is nametag” essentially. So, what it does is it offers people’s names for you in augmented space. So, when you’re using the app, you actually can look around and see everyone’s names hovering over their heads.

Rebecca It’s not a little like Big Brother or anything [Laughter]

John But… it’s an opt-in program.

Renée: Right, it’s an opt-in program. You have to be signed in… and so the target audience is really specific for groups of people who are networking or working together. So, it’s really meant for educational platforms, for workshops, meetings, networking events where you actually want to be interacting and meeting new people. So, really any place where you would be wearing a nametag, this would be an augmented replacement for that, allowing you to see the people in the room from afar or up close, searching for people who maybe you want to make sure you network with, and then having that extra component where… you’re already on your phone… you have this device… you can then connect digitally, you have like this digital business card feature… where you could then connect with them via the app too.

Rebecca So if people want to get involved with Tag AR, what would they need to do?

Renée: Well, at the time of this recording, it will be launched very very soon, so by the time this comes out it should be in the Apple Store and, it is available for download on all iOS devices, on the iPhone. You just have iOS 11 installed and you have to have an A9 processing chip or higher on your phone to experience the augmented experience… so that’s iPhone 6s and above.

Rebecca Great!

Renée: And it’s free.

Rebecca Even better.. [Laughter] Have you designed any other or been involved in any other augmented reality experience development ?

Renée: Yes, so I’ve been actually working and collaborating with different people and different groups on some other projects at the same time as getting Tag AR up and running. I currently am teaching a class called immersive design, which is focused on augmented reality and I’ve been the creative director heading up the project that they’re creating, which is actually a translator app, but it translates… instead of language, it’s actually translating culture. It’s a 3D object recognition application that then translates that… and the target audience for that app is specifically refugees all over the world, who have been displaced from their homes for various reasons, but are trying to familiarize themselves in new culture… and so what it does is it helps actually scan 3D objects, identifies the name and that’s the augmented reality experience and then it uses resources on the web for them to learn how to use that within the culture and save collections of their words that they use most frequently to help them teach the language in the culture.

Rebecca I remember hearing a story about Syracuse, not that long ago… about refugees…. and one of the things that some refugees were struggling with was having electric stoves and knowing what they were and how they worked. So I can imagine… to someone who’s not a refugee or isn’t familiar with those communities, it would be like “I wouldn’t understand why that would be useful,” but I know of some of these really specific stories where “I don’t know what this device is, I have no idea how to use it because we were living in a tent, like we didn’t have an electric stove.”

Renée: Right, and actually the Syracuse community has a lot of refugees… and not a lot of support necessarily in some areas… and so one of those is obviously the cultural changes that they just need, that extra support that my students are helping to… at least help a little bit with some of that culture shock.

Rebecca Does that project have a timeline associated with it?

Renée: Yes, so by the end of the semester it has to be released, and it will be for iOS devices and then over the summer will be developed for Android as well.

Rebecca Great! Were the students doing the programming for that?

Renée: Yeah, students are doing everything. So we’ve taught them the entire experience, so from concept ideation all the way through designing, prototyping, developing and now they’re onto user testing. So, they’re getting the full experience… as they should.

John Are these undergraduates or graduate students?

Renée: It’s actually a mix. So, this is actually an experimental course. It’s the first time the course has been offered at the University. It’s a combination of undergraduate, graduate, and all different majors. So, we have some with programming backgrounds and some with absolutely none, and they’re all diving in and learning how to program and develop mobile apps and create AR experiences. So, it’s been pretty fun…

We’re also working on a few other ones based off of interest and also just some research projects that I have going on. I have a research project called “Augmented Learning” that I’m working on and it’s basically looking at how we can teach tools within the education platform using AR versus the traditional… like if you wanted to learn Adobe Illustrator for instance, you’d have to go from like a Lynda.com video frame to then going back to Illustrator and then going back and forth…and so what it does is… this research project that I’m implementing over the summer and then I’ll be testing and researching in the fall, having students compare student’s learning outcomes based off of augmented learning versus just your traditional platforms. We’re looking at time and how the timeliness is affected because of the Augmented learning experience, so I have that in the works.

Rebecca Sounds really interesting.

Renée: Yes, it should be very interesting. It’s really just waiting on the technology to catch up with my idea….[Laughter]……. Really… that’s what I’m waiting for.

Rebecca I’m noticing a pattern.

Renée:Yes, and so then the other idea that I’m working on is looking at almost like a closed captioning option for students… and the core of all my work is looking at how augmented reality can help overcome learning disabilities. So, Tag AR has an underlining assistance for those who specifically are dyslexic. So by offering a visual, to usually only an auditory component, it allows for additional resources for people (specifically with dyslexia) to have assistance that they need without really making it obvious that it’s specifically for people with learning disabilities… and so I also look to see how AR can help within the classroom setting for people with learning disabilities, but also people who… maybe English is their second language or additional other ways and so this is almost like a closed captioning option. So people could experience the same classroom setting, but they’ll almost like see your closed captioning, like you would see in a television but you would see that in your AR view…

John …in real time.

Renée: In real time, yup! So you could have that translate to a different language or it could just be English to English or whatever the case may be, based off of your need and the whole concept is that you’re getting assistance, but it’s discreet. And so a lot of people with learning disabilities they don’t even tell anyone they have learning disabilities. They don’t get the resources because it could come with a negative connotation… and myself considered. So I’ve never sought any assistance for any of my learning disabilities, including dyslexia, and it would basically empower those who have any challenges to get assistance without it being even known to others, even the person next to you.

John You gave a talk here earlier today, where you were mentioning that, while on the phone it’s not entirely discreet because you have to hold up the phone in front of your face…

Renée: Yes, it’s a little socially awkward.

John …but, you did mention the possibility of migrating this to various types of glasses that are in the very near horizon.

Renée: Yes, so I’ve been looking for partnerships with different companies that are creating the technology that would make this much more discreet and so obviously one of those things in the forefront is the design, right? You want it to look like normal glasses, it shouldn’t look like…

John Google glass…

Renée: Yeah, it shouldn’t look weird, right? Because then it’s very clear that you’re wearing something that’s an assistive device. Actually there’s a company in Rochester, who I’ve been working with that is really great: one, because of their proximity but also just because of the technology and their form and function of their product. Which will take away the awkwardness of holding up your mobile device and it looks like you’re just wearing glasses and you’re getting the additional assistance where needed.

Rebecca You have any initial research findings related to learning disabilities and augmented reality? Have other people done studies that you’ve been looking at or is this really kind of a new frontier?

Renée: Well, people get a lot of research on learning disabilities… and specifically design or typography for dyslexia and that kind of thing. I’ve been working on that and researching what’s available for those kinds of platforms and then seeing how I can then implement that into the AR space. Part of it has been a lot of research in those fields in what already exists and seeing how I can then take that and apply it to the AR component. A lot of things, specifically typography is a big thing, making it very clear what works best and is most readable, especially on a small mobile device or what will work within the optics, we know wearing glasses.

Rebecca and for an ever changing background that you have no control over…

Renée: Yes, so there’s lots of things you can’t control when it’s AR. Light, for instance, is a huge thing. You don’t know how much light will be in the area where people are using these… especially in an educational platform… it might go from really bright to really dark depending on what the professor is doing and then obviously that becomes harder to design for. So, you have to be prepared for the unexpected. The backgrounds could be really busy or they could be really simple and you really just don’t know. So, you have to have really clear separations between the foreground, background… and being very conscious of that design… especially when you’re dealing with accessibility.

Rebecca A lot of your work focuses on design for good, right and research specifically about learning disabilities. How do you see AR having a social impact?

Renée: Well, I think it’s almost a obligation as a designer to show the power of design for good… because it has so much power to do good. So, I almost see it as something that it’s like “yeah, I’m not gonna just design something for the sake of designing something, I’m going to design something that’s going to have a purpose, right?” and so that purpose could change, but I really see, especially within AR space, it’s this idea of practical augmented reality. You could make dinosaurs go across the street in front of you, right? but why? what’s the purpose? and so by adding that extra element of the why and answering the “why?” you actually can then solve a problem that exists within our society and it would offer additional assistance on top of being a really purposeful and helpful platform to design on. I don’t really necessarily look for the area of design for good, I think it almost is just something I gravitate towards because I am a problem solver and I look at things that I think could be improved through design, because design is that powerful and then finding the right platform to solve those specific issues or problems.

Rebecca Where have you seen students struggling as they’ve been designing for AR?

Renée: Development for sure is huge, especially if they’ve never developed before, but the first initial concept is really… if they’ve never designed within 3D space it’s kind of getting the idea of depth in their work, that’s kind of been the biggest challenge initially. Once we do some… just simple prototyping…. and actually I’ve been having them work in After Effects first, before they get into coding, just because that’s how I got into it and to see how the idea of Z space and depth applies from something that really is 2D to something that is 3D… so, taking what they know about 2D, applying 3D to it, and then making that fully immersive jump.

Rebecca So, in After Effects it allows the students to have video which simulates that regular field of vision and then you have your graphics or whatever layering on top.

Renée: Yes, and then adding cameras or giving that prototype feel, so that they can visualize the experience first, before they design it in a place where they have really no control over the environment…. just to give them that practice run.

John For someone who is interested in programming AR apps, what would they need to learn or what types of tools would they need to know?

Renée: So, it kind of depends on what kind of experience you’re creating. So if it’s going to be more of a 3D based object-oriented app, then right now it would be learning Unity…. it’s a little bit of a clunky program…

John But it’s free.

Renée: But it’s free… yeah, or at least parts of it are free… and so kind of like the industry standard I would say for creating those kinds of objects. But, what I’ve been teaching specifically in class, just because of the accessibility and the mainstream effect, would be just programming within Xcode, which is using a language called Swift and it’s actually the most approachable language I’ve ever had to learn… I guess because Apple created it.

John And that’s what most development is in apple.

Renée: Yes, right and so because Apple has their hand on it they usually try to make it a little bit more design friendly. You can definitely see the effect of Apple’s hand on that for sure. It makes it a little easier to teach and students usually can grasp it faster than other programming languages I’ve seen them try to tackle. So I think it really depends on your platform how you want to get the AR experience out. If you really just want to create an AR experience then Unity would work, but if you want it to be something that people can download and interact with on your phone then you need to have it out on a mobile device… so you could use Xcode specifically getting that out on the Apple Store, but Google just came out with their came out of beta for their ARCore so all the Android devices out there will be catching up too.

Rebecca Can you talk about other ways that augmented reality could be used to help aid the learning experience or any existing apps that you’re aware of that that already start moving forward in that direction?

Renée: Yes. So there is an app… I believe it’s called meta. It’s basically a really easy way for people to create AR experiences without knowing any code, and it’s specifically for educational purposes. It basically uses an application on your phone that you’re already clicking… dragging… all those kinds of things you’re used to doing on a phone to create an AR experience. Part of the hard thing with that is obviously the practicality of it. You’re limited to what you can do but I could see some platform resources where you could just very simply, especially for purposes of education, create a quick experience just to help people learn. Obviously the more immersive your teaching is the faster they’re gonna learn it right? So it’s more hands-on and that’s what their their goal is with that app as well, and it’s free. So that’s great too.

Rebecca So an example of using that platform might be if you’re taking students on a tour and you’re trying to get them to think about what it was like in history… a certain period of time… they could you know aim their phone at a particular location or something, right? …and it it could show a picture of what it looked like at a different time or something like that.

Renée: Yes, yep. Actually they have demos of that. So, yeah, that’s a great example.

John One of our colleagues at Fredonia who gave some workshops here. She hasn’t been on the podcast yet, had students in a Freshman Seminar do a Wikitude layer where they created information about various places on campus… where student reviews of them would pop up on Wikitude.

Renée: Great. Yeah, absolutely… and Yelp is kind of doing a similar thing now as well. So as you’re walking around of course you can see all their reviews right over the restaurants as you’re about to go into them, which could help….

Rebecca …which people who are herd people….

Renée: But yeah… It’s kind of a similar idea or concept, right? …of that immersive information layer that can be really helpful as you’re walking around navigating.

John A lot of apps use at least some level of augmented reality, so a lot of people aren’t really aware they’re doing it when they’re looking at Yelp or when they’re looking when they’re searching for things on maps or other things.

Renée: Yes, actually that’s a big thing when people will say, “I don’t know if I am ready for augmented reality or I don’t know how I feel about that.” Part of my response is a lot of people are already using augmented reality… they just don’t realize it… and actually that’s the best part about technology being used well, is if it’s invisible, right? and you don’t even notice that you’re using something and you wouldn’t even consider that AR because it’s just something that feels so natural, and that’s obviously a goal as a designer for sure.

John One less visual one that deals with sound something you had talked about this morning is the Zombie Run app I think it’s still out there I know some people who use that where you can hear zombies approaching spatially to encourage you to move faster or slower and so forth.

Rebecca That sounds terrifying.

John Well, there you go! But….

Renée: I think that’s what they want.

John I’ve heard some people find it motivating, especially fans of The Walking Dead I think.

Renée: But yes, absolutely… audio is a new component that is definitely going to add to the whole AR experience, right? Anything dealing with the senses and especially with exploration of auditory versus visual and how that sensory processing works. The audio component is very important and needs to be also at the forefront in consideration when you’re thinking about these things… especially with wearable devices. They’re going to be much more integrated in the technology to make the audio very clear in the direction and have control over that while at the same time being able to listen to the sounds in your environment… provides a lot of opportunity.

John And you mentioned that you had just looked at some demos of Bose.

Renée: Yes. I was just at South by Southwest… speaking down there, and got to team up with Bose AR and checked out all their 3d prototypes of their AR sunglasses in their wearables and they have some really cool things going on and looking forward to further conversations with them on that so.

John What might be some other applications of AR software for instructional use? Where assignments could be given and students work with AR materials or develop materials?

Renée: Well, the beauty of AR is that its hands-on and its immersive. So, anywhere that someone could in a normal situation where you wouldn’t necessarily be able to have an object in front of you, or you wouldn’t have an experience that you could have ever experienced, because of location or whatever the case maybe… AR provides that opportunity. So, there are some AR apps out there currently but even like thinking about youths and education… thinking about some of the STEM programs and trying to get people understanding how specific things work and how you could build specific things, I think there’s a huge opportunity for AR to help in a space where you get the information right where you need it.

John So, just-in-time instruction and assistance

Renée: Yes.

John Which is similar to the project you were thinking of working on or you’re planning to work on.

Renée: Yes, yep. In augmented learning. Yup.

Rebecca What are you gonna do next?

Renée: Sleep? [LAUGHTER] I have a couple things on the horizon… the biggest thing has been my augmented learning project. I’m really excited to see how I can implement that into my specific design curriculum and then once I see the benefits or things that need to be changed from that… seeing how that I could then have impact not just specific to design but curriculum and the way we learn and the hands-on learning, which I’m a huge advocate for. So, seeing how that can impact the future but also how we can make it a little bit more approachable and kind of getting over those learning curves of the technology to make it really something that can have impact on the way students learn

Rebecca Do you have collaborators? Do you have people that are gonna help you measure some of that?

Renée: Currently for that I have some grant funding that I’m working on to get me started but my hope is that as I keep working on it more I’ll get some more assistance. That is something I’m looking for: Is people who are interested in collaborating as well as making sure I have all the technology and everything needed to have the most user testing that we can have.

John Great.

Rebecca Well, thanks so much for spending some extra time on campus today with us.

Renée: Yes, thanks for having me, and thanks for the tea.

John If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer.