Uptime® magazine recently met with Stéphane Côté, Fellow and Research Director in Bentley Systems’ Applied Research Group. Côté’s work explores the potential of augmented reality in the context of infrastructure engineering. Last spring, he and his team experimented with Microsoft HoloLensTM, an augmented reality device that will shape how all of us will work going forward. Côté talked with us about the HoloLens itself, the benefits of implementing it, and much more.
Augmented reality is a touchstone of any film that is set in the future. The character stands in front of multiple holographic screens, able to access any information he or she wants with the flick of a wrist.
Q: A lot of our audience has never used anything like this before. Can you give us a little background of what the Microsoft HoloLens is and how it works?
A: What the HoloLens can do, essentially, is take the data you want to view and put it in context. I have difficulty remembering people’s names, so the perfect augmented reality application I would like to have is one where I put on my HoloLens and when I see you, your name would appear next to your head. This is one example. Another example is when I go to the grocery store and put a can of food in my cart. How do I know it is good for me? The HoloLens could display extra information related to nutrition, whether people liked it on Facebook® and those sorts of things. Two weeks ago, I was visiting my stepparents and had to do some work in the basement that included destroying an old window and laying out bricks to close the space. I’ve never laid out bricks, so I looked on the Web to learn how to do it. Then, I had to remember the information. I went downstairs and, based on my reconstruction of the video, I laid out bricks. But the augmented reality system would put it all in front of you and show you how to do something, like lay out bricks. It is just there in context, in 3D, exactly where the work has to take place. That’s what makes it so convenient. And, that’s what this whole idea of augmented reality is all about. You can display things at their appropriate place.
Q: What are the types of benefits that we’re going to see, especially in the manufacturing industry and day-to-day operations?
A: Anytime you need to access information, like the specifications of a device, how to do this type of operation, who is the last one that changed that part, that information is stored with a reference to that object. So, you see a pump and it’s broken. And you ask, “Has it been fixed several times?” Using augmented reality, you would just click on it with your finger and all this information would appear in the form of a menu. You could have access to the specifications of that device, the number of times it’s been repaired, the company that installed it, the company that built it and the new models that are available. All that would be next to it. You wouldn’t have to search anymore or browse documents.
Let me explain it another way. Say you have a certain type of maintenance done in your plant every month. It’s always the same guy who does it. Well, one day, he is out sick. No one else knows how to do it, so how can you proceed? Well, one possibility is you send someone with an augmented reality system that gives instructions on how to perform the maintenance. It might say, “I know where you are because I can sense your location. Turn this handle left.” And it’s very clear. There’s an arrow that displays around the handle. You follow the steps and at the end, you’ve done something you’ve never done before. Now, you might need some help in the process. For example, the system says open the handle, but it doesn’t open. You can call someone and that person remotely sees what you’re seeing. And, that person can annotate what you’re seeing. In the case of our example, this person might say, “Okay, you’re using this handle. There is a screw there.” The individual can illustrate this by putting an arrow next to it, so you can see what the person is showing you. Augmented reality would dramatically facilitate collaboration.
Augmented reality also can be used in unexpected situations. Let’s say a pipe bursts or an instrument breaks. You can’t program a HoloLens to tell you how to proceed because you don’t know how to proceed. There’s no written procedure for fixing that. How do you solve that? Instead of using the HoloLens as a teacher or guide, you use it as an assistant. You are in front of the device that’s broken and you ask, “What’s the problem?” The system will be programmed to tell you, “The problem is that instrument doesn’t seem to work anymore.” So, you have to make a decision. You say, “Could you show me the historical values measured by that device?” The system would show you in front of your eyes the graphs of the variations of pressure and then you can browse through that. This could make a huge difference in terms of efficiency.
Q: As I hear about this technology, I think, “Why wouldn’t you want to use this?” It seems so practical and easy, and it eliminates a lot of human error and unnecessary steps. But, is there any resistance to using augmented reality from the current workforce? Or have many in the industry embraced it?
A: The only negative comment I’ve heard is, “Okay, that’s interesting, but that’s not the way we do things currently.” I think one can easily overcome that problem. The augmentation program can be adapted to existing procedures. And, employees can be educated to the new system. It’s not like 20 years ago when computers were still a new thing. I saw how people reacted to the HoloLens. You see, in the center, you have a small dot. That’s your mouse pointer. Now, you must click by doing this. They instantly got the idea in a few seconds. Augmented reality offers a user interface that is very natural to use.
Q: It almost sounds as if somebody is exposed to it; as they try it, they can see it. It’s not a hard technology to understand. It’s just adapting it, like you said, adopting it into their process.
A: Exactly, and I’ll give you an example. The main difficulty of augmented reality is tracking. That is, measuring the position of the tablet or the HoloLens. And it’s extremely important because that’s how we display things at the right place. Let’s say on this table you want to display a virtual pot of flowers. If it was a true object, as you walk around the table, you would expect this pot of flowers to stay in the same place. That’s how real objects behave. But, if you don’t do a good job of tracking, this pot of flowers will move around. And then you’d say, “That’s not a real one because it moves strangely.” What’s so nice about the HoloLens is that the tracking aspect has been mastered in a nearly perfect way. If you put a hologram in the middle of the room or even in the middle of the air, you leave it there and walk around. It behaves as if it were staying there, stable, not moving. It behaves like the physical world. If the augmentation is done well, then people will automatically see these augmentations as normal and they will interact with them as if they were interacting with the real object by taking them and moving them around. It is as if augmented reality is just an extension of reality. It’s very easy to master. You don’t even need to learn to use a map.
Q: That’s really interesting. Where do you see this technology going? I mean, it’s so complete at this point, but what are the possibilities with this technology?
A: It’s just beginning. I was viewing a nine minute movie produced four years ago. It’s called “Sight.” It is a futuristic movie, where Sight is an augmented reality system running on your contact lenses. You see virtual objects, real objects, information, names of people. It’s perfect. And then the lady says during a conversation, “The other day, my Sight crashed.” And the guy in front of her, who is the developer of Sight, says, “No, Sight never crashes.” “Yeah,” she says, “I’m positive my Sight crashed. I was running in the park and it just stopped working. I couldn’t see anything. I was afraid.” Of course, she could see the physical world. She just could not see the augmentations that she was used to seeing with the physical world, like the proximity of a shop, someone suspicious approaching, information about her current e-mails, or all that stuff that would be normal in your everyday life. She was afraid because those were no longer visible. We might become as dependent on augmented reality as we are today with electric power. We feel we can’t do anything if there’s no power.
Now, what was once something that only could be seen in the movies is reality.
Q: That’s an impressive statement—a future you can visualize.
A: What I’ve often heard is that in the future, forgetting your augmented reality device at home would be as bad as it is today when you forget your smartphone. It would be integrated into our lives.
Q: There’s a component that was discussed about the voice-based virtual assistant. Can you explain that a little bit?
A: A worker on-site is typically carrying or using tools. That worker’s hands must be free to do work. There are two ways to interact with augmented reality technology, either by voice or by hand. With your hands, there are only a limited number of gestures you can make and it becomes tiring in the end. So, the voice-enabled technology might be a good choice. Let’s say you’re being taught by a human about how to do something. This person explains, you can ask questions and it’s very natural. That’s what we’re trying to do in our demo, simulate someone who is trying to help you.
Q: It’s just like talking to the machine itself and getting those answers.
Q: This has been very interesting. Is there anything else you would like to add?
A: There are two things. First, this technology, like any other technology, has good and bad aspects. But, if this technology helps you learn faster, become a better person and better understand how to live with other people, then it’s good for us. So, it’s a good idea to start right away and think of ways this technology can help us become better. Secondly, whether we want it or not, whether we think this technology is good or not, it is going to happen and become an integral part of our lives. I believe everything in the future will be augmented everywhere by everyone. These augmentations will give us superpowers – it is like the next step of human evolution.
“R.A.I.” the Reliability.aiTMChatbot
You can ask "R.A.I." anything about maintenance, reliability, and asset management.