Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle

Autonomous underwater vehicle (AUV) plays an increasingly important role in ocean exploration. Existing AUVs are usually not fully autonomous and generally limited to pre-planning or pre-programming tasks. Reinforcement learning (RL) and deep reinforcement learning have been introduced into the AUV...

Full description

Bibliographic Details
Main Authors: Qilei Zhang, Jinying Lin, Qixin Sha, Bo He, Guangliang Li
Format: Article
Language:English
Published: IEEE 2020-01-01
Series:IEEE Access
Subjects:
Online Access:https://ieeexplore.ieee.org/document/8976170/
id doaj-fc08c4ef4e924de4b63b996e99b352d2
record_format Article
spelling doaj-fc08c4ef4e924de4b63b996e99b352d22021-03-30T01:15:19ZengIEEEIEEE Access2169-35362020-01-018242582426810.1109/ACCESS.2020.29704338976170Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater VehicleQilei Zhang0https://orcid.org/0000-0003-3285-8602Jinying Lin1https://orcid.org/0000-0001-8019-9758Qixin Sha2https://orcid.org/0000-0003-0292-3231Bo He3https://orcid.org/0000-0001-6826-4721Guangliang Li4https://orcid.org/0000-0003-1728-5711Department of Electronic Engineering, Ocean University of China, Qingdao, ChinaDepartment of Electronic Engineering, Ocean University of China, Qingdao, ChinaDepartment of Electronic Engineering, Ocean University of China, Qingdao, ChinaDepartment of Electronic Engineering, Ocean University of China, Qingdao, ChinaDepartment of Electronic Engineering, Ocean University of China, Qingdao, ChinaAutonomous underwater vehicle (AUV) plays an increasingly important role in ocean exploration. Existing AUVs are usually not fully autonomous and generally limited to pre-planning or pre-programming tasks. Reinforcement learning (RL) and deep reinforcement learning have been introduced into the AUV design and research to improve its autonomy. However, these methods are still difficult to apply directly to the actual AUV system because of the sparse rewards and low learning efficiency. In this paper, we proposed a deep interactive reinforcement learning method for path following of AUV by combining the advantages of deep reinforcement learning and interactive RL. In addition, since the human trainer cannot provide human rewards for AUV when it is running in the ocean and AUV needs to adapt to a changing environment, we further propose a deep reinforcement learning method that learns from both human rewards and environmental rewards at the same time. We test our methods in two path following tasks-straight line and sinusoids curve following of AUV by simulating in the Gazebo platform. Our experimental results show that with our proposed deep interactive RL method, AUV can converge faster than a DQN learner from only environmental reward. Moreover, AUV learning with our deep RL from both human and environmental rewards can also achieve a similar or even better performance than that with deep interactive RL and can adapt to the actual environment by further learning from environmental rewards.https://ieeexplore.ieee.org/document/8976170/Autonomous underwater vehicleinteractive reinforcement learningdeep Q networkpath following
collection DOAJ
language English
format Article
sources DOAJ
author Qilei Zhang
Jinying Lin
Qixin Sha
Bo He
Guangliang Li
spellingShingle Qilei Zhang
Jinying Lin
Qixin Sha
Bo He
Guangliang Li
Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle
IEEE Access
Autonomous underwater vehicle
interactive reinforcement learning
deep Q network
path following
author_facet Qilei Zhang
Jinying Lin
Qixin Sha
Bo He
Guangliang Li
author_sort Qilei Zhang
title Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle
title_short Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle
title_full Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle
title_fullStr Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle
title_full_unstemmed Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle
title_sort deep interactive reinforcement learning for path following of autonomous underwater vehicle
publisher IEEE
series IEEE Access
issn 2169-3536
publishDate 2020-01-01
description Autonomous underwater vehicle (AUV) plays an increasingly important role in ocean exploration. Existing AUVs are usually not fully autonomous and generally limited to pre-planning or pre-programming tasks. Reinforcement learning (RL) and deep reinforcement learning have been introduced into the AUV design and research to improve its autonomy. However, these methods are still difficult to apply directly to the actual AUV system because of the sparse rewards and low learning efficiency. In this paper, we proposed a deep interactive reinforcement learning method for path following of AUV by combining the advantages of deep reinforcement learning and interactive RL. In addition, since the human trainer cannot provide human rewards for AUV when it is running in the ocean and AUV needs to adapt to a changing environment, we further propose a deep reinforcement learning method that learns from both human rewards and environmental rewards at the same time. We test our methods in two path following tasks-straight line and sinusoids curve following of AUV by simulating in the Gazebo platform. Our experimental results show that with our proposed deep interactive RL method, AUV can converge faster than a DQN learner from only environmental reward. Moreover, AUV learning with our deep RL from both human and environmental rewards can also achieve a similar or even better performance than that with deep interactive RL and can adapt to the actual environment by further learning from environmental rewards.
topic Autonomous underwater vehicle
interactive reinforcement learning
deep Q network
path following
url https://ieeexplore.ieee.org/document/8976170/
work_keys_str_mv AT qileizhang deepinteractivereinforcementlearningforpathfollowingofautonomousunderwatervehicle
AT jinyinglin deepinteractivereinforcementlearningforpathfollowingofautonomousunderwatervehicle
AT qixinsha deepinteractivereinforcementlearningforpathfollowingofautonomousunderwatervehicle
AT bohe deepinteractivereinforcementlearningforpathfollowingofautonomousunderwatervehicle
AT guangliangli deepinteractivereinforcementlearningforpathfollowingofautonomousunderwatervehicle
_version_ 1724187437284982784