Multimodal Interaction Systems Based on Internet of Things and Augmented Reality: A Systematic Literature Review
Technology developments have expanded the diversity of interaction modalities that can be used by an agent (either a human or machine) to interact with a computer system. This expansion has created the need for more natural and user-friendly interfaces in order to achieve effective user experience a...
Main Authors: | , , |
---|---|
Format: | Article |
Language: | English |
Published: |
MDPI AG
2021-02-01
|
Series: | Applied Sciences |
Subjects: | |
Online Access: | https://www.mdpi.com/2076-3417/11/4/1738 |
id |
doaj-920689780b834bc996d176dadaaf5fae |
---|---|
record_format |
Article |
spelling |
doaj-920689780b834bc996d176dadaaf5fae2021-02-17T00:00:41ZengMDPI AGApplied Sciences2076-34172021-02-01111738173810.3390/app11041738Multimodal Interaction Systems Based on Internet of Things and Augmented Reality: A Systematic Literature ReviewJoo Chan Kim0Teemu H. Laine1Christer Åhlund2Department of Computer Science, Electrical and Space Engineering, Luleå University of Technology, 93187 Skellefteå, SwedenDepartment of Digital Media, Ajou University, Suwon 16499, KoreaDepartment of Computer Science, Electrical and Space Engineering, Luleå University of Technology, 93187 Skellefteå, SwedenTechnology developments have expanded the diversity of interaction modalities that can be used by an agent (either a human or machine) to interact with a computer system. This expansion has created the need for more natural and user-friendly interfaces in order to achieve effective user experience and usability. More than one modality can be provided to an agent for interaction with a system to accomplish this goal, which is referred to as a multimodal interaction (MI) system. The Internet of Things (IoT) and augmented reality (AR) are popular technologies that allow interaction systems to combine the real-world context of the agent and immersive AR content. However, although MI systems have been extensively studied, there are only several studies that reviewed MI systems that used IoT and AR. Therefore, this paper presents an in-depth review of studies that proposed various MI systems utilizing IoT and AR. A total of 23 studies were identified and analyzed through a rigorous systematic literature review protocol. The results of our analysis of MI system architectures, the relationship between system components, input/output interaction modalities, and open research challenges are presented and discussed to summarize the findings and identify future research and development avenues for researchers and MI developers.https://www.mdpi.com/2076-3417/11/4/1738multimodal interactioninteraction modalitiesmixed/augmented realityinternet of thingssystematic literature reviewmultimodal UI |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Joo Chan Kim Teemu H. Laine Christer Åhlund |
spellingShingle |
Joo Chan Kim Teemu H. Laine Christer Åhlund Multimodal Interaction Systems Based on Internet of Things and Augmented Reality: A Systematic Literature Review Applied Sciences multimodal interaction interaction modalities mixed/augmented reality internet of things systematic literature review multimodal UI |
author_facet |
Joo Chan Kim Teemu H. Laine Christer Åhlund |
author_sort |
Joo Chan Kim |
title |
Multimodal Interaction Systems Based on Internet of Things and Augmented Reality: A Systematic Literature Review |
title_short |
Multimodal Interaction Systems Based on Internet of Things and Augmented Reality: A Systematic Literature Review |
title_full |
Multimodal Interaction Systems Based on Internet of Things and Augmented Reality: A Systematic Literature Review |
title_fullStr |
Multimodal Interaction Systems Based on Internet of Things and Augmented Reality: A Systematic Literature Review |
title_full_unstemmed |
Multimodal Interaction Systems Based on Internet of Things and Augmented Reality: A Systematic Literature Review |
title_sort |
multimodal interaction systems based on internet of things and augmented reality: a systematic literature review |
publisher |
MDPI AG |
series |
Applied Sciences |
issn |
2076-3417 |
publishDate |
2021-02-01 |
description |
Technology developments have expanded the diversity of interaction modalities that can be used by an agent (either a human or machine) to interact with a computer system. This expansion has created the need for more natural and user-friendly interfaces in order to achieve effective user experience and usability. More than one modality can be provided to an agent for interaction with a system to accomplish this goal, which is referred to as a multimodal interaction (MI) system. The Internet of Things (IoT) and augmented reality (AR) are popular technologies that allow interaction systems to combine the real-world context of the agent and immersive AR content. However, although MI systems have been extensively studied, there are only several studies that reviewed MI systems that used IoT and AR. Therefore, this paper presents an in-depth review of studies that proposed various MI systems utilizing IoT and AR. A total of 23 studies were identified and analyzed through a rigorous systematic literature review protocol. The results of our analysis of MI system architectures, the relationship between system components, input/output interaction modalities, and open research challenges are presented and discussed to summarize the findings and identify future research and development avenues for researchers and MI developers. |
topic |
multimodal interaction interaction modalities mixed/augmented reality internet of things systematic literature review multimodal UI |
url |
https://www.mdpi.com/2076-3417/11/4/1738 |
work_keys_str_mv |
AT joochankim multimodalinteractionsystemsbasedoninternetofthingsandaugmentedrealityasystematicliteraturereview AT teemuhlaine multimodalinteractionsystemsbasedoninternetofthingsandaugmentedrealityasystematicliteraturereview AT christerahlund multimodalinteractionsystemsbasedoninternetofthingsandaugmentedrealityasystematicliteraturereview |
_version_ |
1724265922028371968 |