Deep Learning for Intake Gesture Detection From Wrist-Worn Inertial Sensors: The Effects of Data Preprocessing, Sensor Modalities, and Sensor Positions
Wrist-worn inertial measurement units have emerged as a promising technology to passively capture dietary intake data. State-of-the-art approaches use deep neural networks to process the collected inertial data and detect characteristic hand movements associated with intake gestures. In order to cla...
Main Authors: | , , , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
IEEE
2020-01-01
|
Series: | IEEE Access |
Subjects: | |
Online Access: | https://ieeexplore.ieee.org/document/9187203/ |
id |
doaj-95d52c3ec1e54c29aaca602003037d41 |
---|---|
record_format |
Article |
spelling |
doaj-95d52c3ec1e54c29aaca602003037d412021-03-30T04:01:28ZengIEEEIEEE Access2169-35362020-01-01816493616494910.1109/ACCESS.2020.30220429187203Deep Learning for Intake Gesture Detection From Wrist-Worn Inertial Sensors: The Effects of Data Preprocessing, Sensor Modalities, and Sensor PositionsHamid Heydarian0https://orcid.org/0000-0002-9824-5828Philipp V. Rouast1https://orcid.org/0000-0003-2282-8178Marc T. P. Adam2https://orcid.org/0000-0002-6036-4282Tracy Burrows3Clare E. Collins4Megan E. Rollo5School of Electrical Engineering and Computing, Faculty of Engineering and Built Environment, The University of Newcastle, Callaghan, NSW, AustraliaPriority Research Centre for Physical Activity and Nutrition, The University of Newcastle, Callaghan, NSW, AustraliaSchool of Electrical Engineering and Computing, Faculty of Engineering and Built Environment, The University of Newcastle, Callaghan, NSW, AustraliaPriority Research Centre for Physical Activity and Nutrition, The University of Newcastle, Callaghan, NSW, AustraliaPriority Research Centre for Physical Activity and Nutrition, The University of Newcastle, Callaghan, NSW, AustraliaPriority Research Centre for Physical Activity and Nutrition, The University of Newcastle, Callaghan, NSW, AustraliaWrist-worn inertial measurement units have emerged as a promising technology to passively capture dietary intake data. State-of-the-art approaches use deep neural networks to process the collected inertial data and detect characteristic hand movements associated with intake gestures. In order to clarify the effects of data preprocessing, sensor modalities, and sensor positions, we collected and labeled inertial data from wrist-worn accelerometers and gyroscopes on both hands of 100 participants in a semi-controlled setting. The method included data preprocessing and data segmentation, followed by a two-stage approach. In Stage 1, we estimated the probability of each inertial data frame being intake or non-intake, benchmarking different deep learning models and architectures. Based on the probabilities estimated in Stage 1, we detected the intake gestures in Stage 2 and calculated the F1 score for each model. Results indicate that top model performance was achieved by a CNN-LSTM with earliest sensor data fusion through a dedicated CNN layer and a target matching technique (F<sub>1</sub> = .778). As for data preprocessing, results show that applying a consecutive combination of mirroring, removing gravity effect, and standardization was beneficial for model performance, while smoothing had adverse effects. We further investigate the effectiveness of using different combinations of sensor modalities (i.e., accelerometer and/or gyroscope) and sensor positions (i.e., dominant intake hand and/or non-dominant intake hand).https://ieeexplore.ieee.org/document/9187203/Accelerometerdeep learningintake gesture detectiongyroscopewrist-worn |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Hamid Heydarian Philipp V. Rouast Marc T. P. Adam Tracy Burrows Clare E. Collins Megan E. Rollo |
spellingShingle |
Hamid Heydarian Philipp V. Rouast Marc T. P. Adam Tracy Burrows Clare E. Collins Megan E. Rollo Deep Learning for Intake Gesture Detection From Wrist-Worn Inertial Sensors: The Effects of Data Preprocessing, Sensor Modalities, and Sensor Positions IEEE Access Accelerometer deep learning intake gesture detection gyroscope wrist-worn |
author_facet |
Hamid Heydarian Philipp V. Rouast Marc T. P. Adam Tracy Burrows Clare E. Collins Megan E. Rollo |
author_sort |
Hamid Heydarian |
title |
Deep Learning for Intake Gesture Detection From Wrist-Worn Inertial Sensors: The Effects of Data Preprocessing, Sensor Modalities, and Sensor Positions |
title_short |
Deep Learning for Intake Gesture Detection From Wrist-Worn Inertial Sensors: The Effects of Data Preprocessing, Sensor Modalities, and Sensor Positions |
title_full |
Deep Learning for Intake Gesture Detection From Wrist-Worn Inertial Sensors: The Effects of Data Preprocessing, Sensor Modalities, and Sensor Positions |
title_fullStr |
Deep Learning for Intake Gesture Detection From Wrist-Worn Inertial Sensors: The Effects of Data Preprocessing, Sensor Modalities, and Sensor Positions |
title_full_unstemmed |
Deep Learning for Intake Gesture Detection From Wrist-Worn Inertial Sensors: The Effects of Data Preprocessing, Sensor Modalities, and Sensor Positions |
title_sort |
deep learning for intake gesture detection from wrist-worn inertial sensors: the effects of data preprocessing, sensor modalities, and sensor positions |
publisher |
IEEE |
series |
IEEE Access |
issn |
2169-3536 |
publishDate |
2020-01-01 |
description |
Wrist-worn inertial measurement units have emerged as a promising technology to passively capture dietary intake data. State-of-the-art approaches use deep neural networks to process the collected inertial data and detect characteristic hand movements associated with intake gestures. In order to clarify the effects of data preprocessing, sensor modalities, and sensor positions, we collected and labeled inertial data from wrist-worn accelerometers and gyroscopes on both hands of 100 participants in a semi-controlled setting. The method included data preprocessing and data segmentation, followed by a two-stage approach. In Stage 1, we estimated the probability of each inertial data frame being intake or non-intake, benchmarking different deep learning models and architectures. Based on the probabilities estimated in Stage 1, we detected the intake gestures in Stage 2 and calculated the F1 score for each model. Results indicate that top model performance was achieved by a CNN-LSTM with earliest sensor data fusion through a dedicated CNN layer and a target matching technique (F<sub>1</sub> = .778). As for data preprocessing, results show that applying a consecutive combination of mirroring, removing gravity effect, and standardization was beneficial for model performance, while smoothing had adverse effects. We further investigate the effectiveness of using different combinations of sensor modalities (i.e., accelerometer and/or gyroscope) and sensor positions (i.e., dominant intake hand and/or non-dominant intake hand). |
topic |
Accelerometer deep learning intake gesture detection gyroscope wrist-worn |
url |
https://ieeexplore.ieee.org/document/9187203/ |
work_keys_str_mv |
AT hamidheydarian deeplearningforintakegesturedetectionfromwristworninertialsensorstheeffectsofdatapreprocessingsensormodalitiesandsensorpositions AT philippvrouast deeplearningforintakegesturedetectionfromwristworninertialsensorstheeffectsofdatapreprocessingsensormodalitiesandsensorpositions AT marctpadam deeplearningforintakegesturedetectionfromwristworninertialsensorstheeffectsofdatapreprocessingsensormodalitiesandsensorpositions AT tracyburrows deeplearningforintakegesturedetectionfromwristworninertialsensorstheeffectsofdatapreprocessingsensormodalitiesandsensorpositions AT clareecollins deeplearningforintakegesturedetectionfromwristworninertialsensorstheeffectsofdatapreprocessingsensormodalitiesandsensorpositions AT meganerollo deeplearningforintakegesturedetectionfromwristworninertialsensorstheeffectsofdatapreprocessingsensormodalitiesandsensorpositions |
_version_ |
1724182539812208640 |