Memory-Based Deep Neural Attention (mDNA) for Cognitive Multi-Turn Response Retrieval in Task-Oriented Chatbots
One of the important criteria used in judging the performance of a chatbot is the ability to provide meaningful and informative responses that correspond with the context of a user’s utterance. Nowadays, the number of enterprises adopting and relying on task-oriented chatbots for profit is increasin...
Main Authors: | , , |
---|---|
Format: | Article |
Language: | English |
Published: |
MDPI AG
2020-08-01
|
Series: | Applied Sciences |
Subjects: | |
Online Access: | https://www.mdpi.com/2076-3417/10/17/5819 |
id |
doaj-4fbba14888b440f5a7dea017391b7380 |
---|---|
record_format |
Article |
spelling |
doaj-4fbba14888b440f5a7dea017391b73802020-11-25T03:55:04ZengMDPI AGApplied Sciences2076-34172020-08-01105819581910.3390/app10175819Memory-Based Deep Neural Attention (mDNA) for Cognitive Multi-Turn Response Retrieval in Task-Oriented ChatbotsJenhui Chen0Obinna Agbodike1Lei Wang2Department of Computer Science and Information Engineering, Chang Gung University, Kweishan, Taoyuan 33302, TaiwanDepartment of Electrical Engineering, Chang Gung University, Kweishan, Taoyuan 33302, TaiwanSchool of Software, Dalian University of Technology, Dalian 116024, ChinaOne of the important criteria used in judging the performance of a chatbot is the ability to provide meaningful and informative responses that correspond with the context of a user’s utterance. Nowadays, the number of enterprises adopting and relying on task-oriented chatbots for profit is increasing. Dialog errors and inappropriate response to user queries by chatbots can result in huge cost implications. To achieve high performance, recent AI chatbot models are increasingly adopting the Transformer positional encoding and the attention-based architecture. While the transformer performs optimally in sequential generative chatbot models, recent studies has pointed out the occurrence of logical inconsistency and fuzzy error problems when the Transformer technique is adopted in retrieval-based chatbot models. Our investigation discovers that the encountered errors are caused by information losses. Therefore, in this paper, we address this problem by augmenting the Transformer-based retrieval chatbot architecture with a memory-based deep neural attention (mDNA) model by using an approach similar to late data fusion. The mDNA is a simple encoder-decoder neural architecture that comprises of bidirectional long short-term memory (Bi-LSTM), attention mechanism, and a memory for information retention in the encoder. In our experiments, we trained the model extensively on a large Ubuntu dialog corpus, and the results from recall evaluation scores show that the mDNA augmentation approach slightly outperforms selected state-of-the-art retrieval chatbot models. The results from the mDNA augmentation approach are quite impressive.https://www.mdpi.com/2076-3417/10/17/5819Bi-LSTMmemoryNLPattentiondialog-systemretrieval |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Jenhui Chen Obinna Agbodike Lei Wang |
spellingShingle |
Jenhui Chen Obinna Agbodike Lei Wang Memory-Based Deep Neural Attention (mDNA) for Cognitive Multi-Turn Response Retrieval in Task-Oriented Chatbots Applied Sciences Bi-LSTM memory NLP attention dialog-system retrieval |
author_facet |
Jenhui Chen Obinna Agbodike Lei Wang |
author_sort |
Jenhui Chen |
title |
Memory-Based Deep Neural Attention (mDNA) for Cognitive Multi-Turn Response Retrieval in Task-Oriented Chatbots |
title_short |
Memory-Based Deep Neural Attention (mDNA) for Cognitive Multi-Turn Response Retrieval in Task-Oriented Chatbots |
title_full |
Memory-Based Deep Neural Attention (mDNA) for Cognitive Multi-Turn Response Retrieval in Task-Oriented Chatbots |
title_fullStr |
Memory-Based Deep Neural Attention (mDNA) for Cognitive Multi-Turn Response Retrieval in Task-Oriented Chatbots |
title_full_unstemmed |
Memory-Based Deep Neural Attention (mDNA) for Cognitive Multi-Turn Response Retrieval in Task-Oriented Chatbots |
title_sort |
memory-based deep neural attention (mdna) for cognitive multi-turn response retrieval in task-oriented chatbots |
publisher |
MDPI AG |
series |
Applied Sciences |
issn |
2076-3417 |
publishDate |
2020-08-01 |
description |
One of the important criteria used in judging the performance of a chatbot is the ability to provide meaningful and informative responses that correspond with the context of a user’s utterance. Nowadays, the number of enterprises adopting and relying on task-oriented chatbots for profit is increasing. Dialog errors and inappropriate response to user queries by chatbots can result in huge cost implications. To achieve high performance, recent AI chatbot models are increasingly adopting the Transformer positional encoding and the attention-based architecture. While the transformer performs optimally in sequential generative chatbot models, recent studies has pointed out the occurrence of logical inconsistency and fuzzy error problems when the Transformer technique is adopted in retrieval-based chatbot models. Our investigation discovers that the encountered errors are caused by information losses. Therefore, in this paper, we address this problem by augmenting the Transformer-based retrieval chatbot architecture with a memory-based deep neural attention (mDNA) model by using an approach similar to late data fusion. The mDNA is a simple encoder-decoder neural architecture that comprises of bidirectional long short-term memory (Bi-LSTM), attention mechanism, and a memory for information retention in the encoder. In our experiments, we trained the model extensively on a large Ubuntu dialog corpus, and the results from recall evaluation scores show that the mDNA augmentation approach slightly outperforms selected state-of-the-art retrieval chatbot models. The results from the mDNA augmentation approach are quite impressive. |
topic |
Bi-LSTM memory NLP attention dialog-system retrieval |
url |
https://www.mdpi.com/2076-3417/10/17/5819 |
work_keys_str_mv |
AT jenhuichen memorybaseddeepneuralattentionmdnaforcognitivemultiturnresponseretrievalintaskorientedchatbots AT obinnaagbodike memorybaseddeepneuralattentionmdnaforcognitivemultiturnresponseretrievalintaskorientedchatbots AT leiwang memorybaseddeepneuralattentionmdnaforcognitivemultiturnresponseretrievalintaskorientedchatbots |
_version_ |
1724470973165469696 |