Manipulation Action Recognition and Reconstruction using a Deep Scene Graph Network

Convolutional neural networks have been successfully used in action recognition but are usually restricted to operate on Euclidean data, such as images. In recent years there has been an increase in research devoted towards finding a generalized model operating on non-Euclidean data (e.g graphs) and...

Full description

Bibliographic Details
Main Authors: Ejdeholm, Dawid, Harsten, Jacob
Format: Others
Language:English
Published: Högskolan i Halmstad, Akademin för informationsteknologi 2020
Subjects:
Online Access:http://urn.kb.se/resolve?urn=urn:nbn:se:hh:diva-42405
id ndltd-UPSALLA1-oai-DiVA.org-hh-42405
record_format oai_dc
spelling ndltd-UPSALLA1-oai-DiVA.org-hh-424052020-06-16T03:32:41ZManipulation Action Recognition and Reconstruction using a Deep Scene Graph NetworkengEjdeholm, DawidHarsten, JacobHögskolan i Halmstad, Akademin för informationsteknologiHögskolan i Halmstad, Akademin för informationsteknologi2020RoboticsRobotteknik och automationConvolutional neural networks have been successfully used in action recognition but are usually restricted to operate on Euclidean data, such as images. In recent years there has been an increase in research devoted towards finding a generalized model operating on non-Euclidean data (e.g graphs) and manipulation action recognition on graphs is still a very novel subject. In this thesis a novel graph based deep neural network is developed for predicting manipulation actions and reconstructing graphs from a lower space representation. The network is trained on two manipulation action datasets and uses their, respective, previous works on action prediction as a baseline. In addition, a modular perception pipeline is developed that takes RGBD images as input and outputs a scene graph, consisting of objects and their spatial relations, which can then be fed to the network to lead to online action prediction. The network manages to outperform both baselines when training for action prediction and achieves comparable results when trained in an end-to-end manner performing both action prediction and graph reconstruction, simultaneously. Furthermore, to test the scalability of our model, the network is tested with input graphs deriving from our scene graph generator where the subject is performing 7 different demonstrations of the learned action types in a new scene context with novel objects. Student thesisinfo:eu-repo/semantics/bachelorThesistexthttp://urn.kb.se/resolve?urn=urn:nbn:se:hh:diva-42405application/pdfinfo:eu-repo/semantics/openAccess
collection NDLTD
language English
format Others
sources NDLTD
topic Robotics
Robotteknik och automation
spellingShingle Robotics
Robotteknik och automation
Ejdeholm, Dawid
Harsten, Jacob
Manipulation Action Recognition and Reconstruction using a Deep Scene Graph Network
description Convolutional neural networks have been successfully used in action recognition but are usually restricted to operate on Euclidean data, such as images. In recent years there has been an increase in research devoted towards finding a generalized model operating on non-Euclidean data (e.g graphs) and manipulation action recognition on graphs is still a very novel subject. In this thesis a novel graph based deep neural network is developed for predicting manipulation actions and reconstructing graphs from a lower space representation. The network is trained on two manipulation action datasets and uses their, respective, previous works on action prediction as a baseline. In addition, a modular perception pipeline is developed that takes RGBD images as input and outputs a scene graph, consisting of objects and their spatial relations, which can then be fed to the network to lead to online action prediction. The network manages to outperform both baselines when training for action prediction and achieves comparable results when trained in an end-to-end manner performing both action prediction and graph reconstruction, simultaneously. Furthermore, to test the scalability of our model, the network is tested with input graphs deriving from our scene graph generator where the subject is performing 7 different demonstrations of the learned action types in a new scene context with novel objects.
author Ejdeholm, Dawid
Harsten, Jacob
author_facet Ejdeholm, Dawid
Harsten, Jacob
author_sort Ejdeholm, Dawid
title Manipulation Action Recognition and Reconstruction using a Deep Scene Graph Network
title_short Manipulation Action Recognition and Reconstruction using a Deep Scene Graph Network
title_full Manipulation Action Recognition and Reconstruction using a Deep Scene Graph Network
title_fullStr Manipulation Action Recognition and Reconstruction using a Deep Scene Graph Network
title_full_unstemmed Manipulation Action Recognition and Reconstruction using a Deep Scene Graph Network
title_sort manipulation action recognition and reconstruction using a deep scene graph network
publisher Högskolan i Halmstad, Akademin för informationsteknologi
publishDate 2020
url http://urn.kb.se/resolve?urn=urn:nbn:se:hh:diva-42405
work_keys_str_mv AT ejdeholmdawid manipulationactionrecognitionandreconstructionusingadeepscenegraphnetwork
AT harstenjacob manipulationactionrecognitionandreconstructionusingadeepscenegraphnetwork
_version_ 1719320238429831168