Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences
We have been investigating rakugo speech synthesis as a challenging example of speech synthesis that entertains audiences. Rakugo is a traditional Japanese form of verbal entertainment similar to a combination of one-person stand-up comedy and comic storytelling and is popular even today. In rakugo,...
Main Authors: | , , , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
IEEE
2020-01-01
|
Series: | IEEE Access |
Subjects: | |
Online Access: | https://ieeexplore.ieee.org/document/9149598/ |
id |
doaj-fcff8e136f1245b0a48ed3885f62205a |
---|---|
record_format |
Article |
spelling |
doaj-fcff8e136f1245b0a48ed3885f62205a2021-03-30T04:19:03ZengIEEEIEEE Access2169-35362020-01-01813814913816110.1109/ACCESS.2020.30119759149598Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains AudiencesShuhei Kato0https://orcid.org/0000-0001-5746-6811Yusuke Yasuda1https://orcid.org/0000-0002-2130-747XXin Wang2https://orcid.org/0000-0001-8246-0606Erica Cooper3https://orcid.org/0000-0002-2978-2793Shinji Takaki4https://orcid.org/0000-0001-7294-7699Junichi Yamagishi5https://orcid.org/0000-0003-2752-3955National Institute of Informatics, The Graduate University for Advanced Sciences (SOKENDAI), Hayama, JapanNational Institute of Informatics, The Graduate University for Advanced Sciences (SOKENDAI), Hayama, JapanNational Institute of Informatics, Chiyoda, JapanNational Institute of Informatics, Chiyoda, JapanNagoya Institute of Technology, Nagoya, JapanNational Institute of Informatics, Chiyoda, JapanWe have been investigating rakugo speech synthesis as a challenging example of speech synthesis that entertains audiences. Rakugo is a traditional Japanese form of verbal entertainment similar to a combination of one-person stand-up comedy and comic storytelling and is popular even today. In rakugo, a performer plays multiple characters, and conversations or dialogues between the characters make the story progress. To investigate how close the quality of synthesized rakugo speech can approach that of professionals' speech, we modeled rakugo speech using Tacotron 2, a state-of-the-art speech synthesis system that can produce speech that sounds as natural as human speech albeit under limited conditions, and an enhanced version of it with self-attention to better consider long-term dependencies. We also used global style tokens and manually labeled context features to enrich speaking styles. Through a listening test, we measured not only naturalness but also distinguishability of characters, understandability of the content, and the degree of entertainment. Although we found that the speech synthesis models could not yet reach the professional level, the results of the listening test provided interesting insights: 1) we should not focus only on the naturalness of synthesized speech but also the distinguishability of characters and the understandability of the content to further entertain audiences; 2) the fundamental frequency (f<sub>o</sub>) expressions of synthesized speech are poorer than those of human speech, and more entertaining speech should have richer f<sub>o</sub> expression. Although there is room for improvement, we believe this is an important stepping stone toward achieving entertaining speech synthesis at the professional level.https://ieeexplore.ieee.org/document/9149598/Contextentertainmentglobal style tokensrakugoself-attentionspeech synthesis |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Shuhei Kato Yusuke Yasuda Xin Wang Erica Cooper Shinji Takaki Junichi Yamagishi |
spellingShingle |
Shuhei Kato Yusuke Yasuda Xin Wang Erica Cooper Shinji Takaki Junichi Yamagishi Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences IEEE Access Context entertainment global style tokens rakugo self-attention speech synthesis |
author_facet |
Shuhei Kato Yusuke Yasuda Xin Wang Erica Cooper Shinji Takaki Junichi Yamagishi |
author_sort |
Shuhei Kato |
title |
Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences |
title_short |
Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences |
title_full |
Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences |
title_fullStr |
Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences |
title_full_unstemmed |
Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences |
title_sort |
modeling of rakugo speech and its limitations: toward speech synthesis that entertains audiences |
publisher |
IEEE |
series |
IEEE Access |
issn |
2169-3536 |
publishDate |
2020-01-01 |
description |
We have been investigating rakugo speech synthesis as a challenging example of speech synthesis that entertains audiences. Rakugo is a traditional Japanese form of verbal entertainment similar to a combination of one-person stand-up comedy and comic storytelling and is popular even today. In rakugo, a performer plays multiple characters, and conversations or dialogues between the characters make the story progress. To investigate how close the quality of synthesized rakugo speech can approach that of professionals' speech, we modeled rakugo speech using Tacotron 2, a state-of-the-art speech synthesis system that can produce speech that sounds as natural as human speech albeit under limited conditions, and an enhanced version of it with self-attention to better consider long-term dependencies. We also used global style tokens and manually labeled context features to enrich speaking styles. Through a listening test, we measured not only naturalness but also distinguishability of characters, understandability of the content, and the degree of entertainment. Although we found that the speech synthesis models could not yet reach the professional level, the results of the listening test provided interesting insights: 1) we should not focus only on the naturalness of synthesized speech but also the distinguishability of characters and the understandability of the content to further entertain audiences; 2) the fundamental frequency (f<sub>o</sub>) expressions of synthesized speech are poorer than those of human speech, and more entertaining speech should have richer f<sub>o</sub> expression. Although there is room for improvement, we believe this is an important stepping stone toward achieving entertaining speech synthesis at the professional level. |
topic |
Context entertainment global style tokens rakugo self-attention speech synthesis |
url |
https://ieeexplore.ieee.org/document/9149598/ |
work_keys_str_mv |
AT shuheikato modelingofrakugospeechanditslimitationstowardspeechsynthesisthatentertainsaudiences AT yusukeyasuda modelingofrakugospeechanditslimitationstowardspeechsynthesisthatentertainsaudiences AT xinwang modelingofrakugospeechanditslimitationstowardspeechsynthesisthatentertainsaudiences AT ericacooper modelingofrakugospeechanditslimitationstowardspeechsynthesisthatentertainsaudiences AT shinjitakaki modelingofrakugospeechanditslimitationstowardspeechsynthesisthatentertainsaudiences AT junichiyamagishi modelingofrakugospeechanditslimitationstowardspeechsynthesisthatentertainsaudiences |
_version_ |
1724182042658209792 |