Unraveling the Contribution of Image Captioning and Neural Machine Translation for Multimodal Machine Translation

Unraveling the Contribution of Image Captioning and Neural Machine Translation for Multimodal Machine Translation

Recent work on multimodal machine translation has attempted to address the problem of producing target language image descriptions based on both the source language description and the corresponding image. However, existing work has not been conclusive on the contribution of visual information. This...

Full description

Bibliographic Details
Main Authors: Lala Chiraag, Madhyastha Pranava, Wang Josiah, Specia Lucia
Format: Article
Language:English
Published: Sciendo 2017-06-01
Series:Prague Bulletin of Mathematical Linguistics
Online Access:https://doi.org/10.1515/pralin-2017-0020
id doaj-112a4b58e0ec45d38063bf134a37730b
record_format Article
spelling doaj-112a4b58e0ec45d38063bf134a37730b2021-09-05T13:59:53ZengSciendoPrague Bulletin of Mathematical Linguistics 1804-04622017-06-01108119720810.1515/pralin-2017-0020pralin-2017-0020Unraveling the Contribution of Image Captioning and Neural Machine Translation for Multimodal Machine TranslationLala Chiraag0Madhyastha Pranava1Wang Josiah2Specia Lucia3University of SheffieldUniversity of SheffieldUniversity of SheffieldUniversity of SheffieldRecent work on multimodal machine translation has attempted to address the problem of producing target language image descriptions based on both the source language description and the corresponding image. However, existing work has not been conclusive on the contribution of visual information. This paper presents an in-depth study of the problem by examining the differences and complementarities of two related but distinct approaches to this task: textonly neural machine translation and image captioning. We analyse the scope for improvement and the effect of different data and settings to build models for these tasks. We also propose ways of combining these two approaches for improved translation quality.https://doi.org/10.1515/pralin-2017-0020
collection DOAJ
language English
format Article
sources DOAJ
author Lala Chiraag
Madhyastha Pranava
Wang Josiah
Specia Lucia
spellingShingle Lala Chiraag
Madhyastha Pranava
Wang Josiah
Specia Lucia
Unraveling the Contribution of Image Captioning and Neural Machine Translation for Multimodal Machine Translation
Prague Bulletin of Mathematical Linguistics
author_facet Lala Chiraag
Madhyastha Pranava
Wang Josiah
Specia Lucia
author_sort Lala Chiraag
title Unraveling the Contribution of Image Captioning and Neural Machine Translation for Multimodal Machine Translation
title_short Unraveling the Contribution of Image Captioning and Neural Machine Translation for Multimodal Machine Translation
title_full Unraveling the Contribution of Image Captioning and Neural Machine Translation for Multimodal Machine Translation
title_fullStr Unraveling the Contribution of Image Captioning and Neural Machine Translation for Multimodal Machine Translation
title_full_unstemmed Unraveling the Contribution of Image Captioning and Neural Machine Translation for Multimodal Machine Translation
title_sort unraveling the contribution of image captioning and neural machine translation for multimodal machine translation
publisher Sciendo
series Prague Bulletin of Mathematical Linguistics
issn 1804-0462
publishDate 2017-06-01
description Recent work on multimodal machine translation has attempted to address the problem of producing target language image descriptions based on both the source language description and the corresponding image. However, existing work has not been conclusive on the contribution of visual information. This paper presents an in-depth study of the problem by examining the differences and complementarities of two related but distinct approaches to this task: textonly neural machine translation and image captioning. We analyse the scope for improvement and the effect of different data and settings to build models for these tasks. We also propose ways of combining these two approaches for improved translation quality.
url https://doi.org/10.1515/pralin-2017-0020
work_keys_str_mv AT lalachiraag unravelingthecontributionofimagecaptioningandneuralmachinetranslationformultimodalmachinetranslation
AT madhyasthapranava unravelingthecontributionofimagecaptioningandneuralmachinetranslationformultimodalmachinetranslation
AT wangjosiah unravelingthecontributionofimagecaptioningandneuralmachinetranslationformultimodalmachinetranslation
AT specialucia unravelingthecontributionofimagecaptioningandneuralmachinetranslationformultimodalmachinetranslation
_version_ 1717812789932720128

Similar Items