Multimodal Warnings in Remote Operation: The Case Study on Remote Driving

• Main Authors: Pekka Kallioniemi, Alisa Burova, John Mäkelä, Tuuli Keskinen, Kimmo Ronkainen, Ville Mäkelä, Jaakko Hakulinen, Markku Turunen
• Format: Article
• Language: English
• Published: MDPI AG 2021-08-01
• Series: Multimodal Technologies and Interaction
• Subjects: remote operations; multimodal feedback; human factors
• Online Access: https://www.mdpi.com/2414-4088/5/8/44

Developments in sensor technology, artificial intelligence, and network technologies like 5G has made remote operation a valuable method of controlling various types of machinery. The benefits of remote operations come with an opportunity to access hazardous environments. The major limitation of remote operation is the lack of proper sensory feedback from the machine, which in turn negatively affects situational awareness and, consequently, may risk remote operations. This article explores how to improve situational awareness via multimodal feedback (visual, auditory, and haptic) and studies how it can be utilized to communicate warnings to remote operators. To reach our goals, we conducted a controlled, within-subjects experiment in eight conditions with twenty-four participants on a simulated remote driving system. Additionally, we gathered further insights with a UX questionnaire and semi-structured interviews. Gathered data showed that the use of multimodal feedback positively affected situational awareness when driving remotely. Our findings indicate that the combination of added haptic and visual feedback was considered the best feedback combination to communicate the slipperiness of the road. We also found that the feeling of presence is an important aspect of remote driving tasks, and a requested one, especially by those with more experience in operating real heavy machinery.