Favorite Video Classification Based on Multimodal Bidirectional LSTM
Video classification based on the user's preference (information of what a user likes: WUL) is important for realizing human-centered video retrieval. A better understanding of the rationale of WUL would greatly contribute to the support for successful video retrieval. However, a few studies ha...
Main Authors: | , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
IEEE
2018-01-01
|
Series: | IEEE Access |
Subjects: | |
Online Access: | https://ieeexplore.ieee.org/document/8496751/ |
id |
doaj-c2d78fd24445421ab4d6f5542242033c |
---|---|
record_format |
Article |
spelling |
doaj-c2d78fd24445421ab4d6f5542242033c2021-03-29T21:21:58ZengIEEEIEEE Access2169-35362018-01-016614016140910.1109/ACCESS.2018.28767108496751Favorite Video Classification Based on Multimodal Bidirectional LSTMTakahiro Ogawa0https://orcid.org/0000-0001-5332-8112Yuma Sasaka1https://orcid.org/0000-0003-0657-0816Keisuke Maeda2Miki Haseyama3Graduate School of Information Science and Technology, Hokkaido University, Sapporo, JapanGraduate School of Information Science and Technology, Hokkaido University, Sapporo, JapanGraduate School of Information Science and Technology, Hokkaido University, Sapporo, JapanGraduate School of Information Science and Technology, Hokkaido University, Sapporo, JapanVideo classification based on the user's preference (information of what a user likes: WUL) is important for realizing human-centered video retrieval. A better understanding of the rationale of WUL would greatly contribute to the support for successful video retrieval. However, a few studies have shown the relationship between information of what a user watches and WUL. A new method that classifies videos on the basis of WUL using video features and electroencephalogram (EEG) signals collaboratively with a multimodal bidirectional Long Short-Term Memory (Bi-LSTM) network is presented in this paper. To the best of our knowledge, there has been no study on WUL-based video classification using video features and EEG signals collaboratively with LSTM. First, we newly apply transfer learning to the WUL-based video classification since the number of labels (liked or not liked) attached to videos by users is small, and it is difficult to classify videos based on WUL. Furthermore, we conduct a user study for showing that the representation of psychophysiological signals calculated from Bi-LSTM is effective for the WUL-based video classification. Experimental results showed that our deep neural network feature representations can distinguish WUL for each subject.https://ieeexplore.ieee.org/document/8496751/Multimodal fusionvideo classificationLSTMEEG |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Takahiro Ogawa Yuma Sasaka Keisuke Maeda Miki Haseyama |
spellingShingle |
Takahiro Ogawa Yuma Sasaka Keisuke Maeda Miki Haseyama Favorite Video Classification Based on Multimodal Bidirectional LSTM IEEE Access Multimodal fusion video classification LSTM EEG |
author_facet |
Takahiro Ogawa Yuma Sasaka Keisuke Maeda Miki Haseyama |
author_sort |
Takahiro Ogawa |
title |
Favorite Video Classification Based on Multimodal Bidirectional LSTM |
title_short |
Favorite Video Classification Based on Multimodal Bidirectional LSTM |
title_full |
Favorite Video Classification Based on Multimodal Bidirectional LSTM |
title_fullStr |
Favorite Video Classification Based on Multimodal Bidirectional LSTM |
title_full_unstemmed |
Favorite Video Classification Based on Multimodal Bidirectional LSTM |
title_sort |
favorite video classification based on multimodal bidirectional lstm |
publisher |
IEEE |
series |
IEEE Access |
issn |
2169-3536 |
publishDate |
2018-01-01 |
description |
Video classification based on the user's preference (information of what a user likes: WUL) is important for realizing human-centered video retrieval. A better understanding of the rationale of WUL would greatly contribute to the support for successful video retrieval. However, a few studies have shown the relationship between information of what a user watches and WUL. A new method that classifies videos on the basis of WUL using video features and electroencephalogram (EEG) signals collaboratively with a multimodal bidirectional Long Short-Term Memory (Bi-LSTM) network is presented in this paper. To the best of our knowledge, there has been no study on WUL-based video classification using video features and EEG signals collaboratively with LSTM. First, we newly apply transfer learning to the WUL-based video classification since the number of labels (liked or not liked) attached to videos by users is small, and it is difficult to classify videos based on WUL. Furthermore, we conduct a user study for showing that the representation of psychophysiological signals calculated from Bi-LSTM is effective for the WUL-based video classification. Experimental results showed that our deep neural network feature representations can distinguish WUL for each subject. |
topic |
Multimodal fusion video classification LSTM EEG |
url |
https://ieeexplore.ieee.org/document/8496751/ |
work_keys_str_mv |
AT takahiroogawa favoritevideoclassificationbasedonmultimodalbidirectionallstm AT yumasasaka favoritevideoclassificationbasedonmultimodalbidirectionallstm AT keisukemaeda favoritevideoclassificationbasedonmultimodalbidirectionallstm AT mikihaseyama favoritevideoclassificationbasedonmultimodalbidirectionallstm |
_version_ |
1724193032303017984 |