Show simple item record

dc.contributor.authorYang, Ziyan
Pinto-Alva, Leticia
Dernoncourt, Franck
Ordonez, Vicente
dc.date.accessioned 2022-03-24T13:31:41Z
dc.date.available 2022-03-24T13:31:41Z
dc.date.issued 2022
dc.identifier.citation Yang, Ziyan, Pinto-Alva, Leticia, Dernoncourt, Franck, et al.. "Backpropagation-Based Decoding for Multimodal Machine Translation." Frontiers in Artificial Intelligence, 4, (2022) Frontiers Media S.A.: https://doi.org/10.3389/frai.2021.736722.
dc.identifier.urihttps://hdl.handle.net/1911/112045
dc.description.abstract People are able to describe images using thousands of languages, but languages share only one visual world. The aim of this work is to use the learned intermediate visual representations from a deep convolutional neural network to transfer information across languages for which paired data is not available in any form. Our work proposes using backpropagation-based decoding coupled with transformer-based multilingual-multimodal language models in order to obtain translations between any languages used during training. We particularly show the capabilities of this approach in the translation of German-Japanese and Japanese-German sentence pairs, given a training data of images freely associated with text in English, German, and Japanese but for which no single image contains annotations in both Japanese and German. Moreover, we demonstrate that our approach is also generally useful in the multilingual image captioning task when sentences in a second language are available at test time. The results of our method also compare favorably in the Multi30k dataset against recently proposed methods that are also aiming to leverage images as an intermediate source of translations.
dc.language.iso eng
dc.publisher Frontiers Media S.A.
dc.rights This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.title Backpropagation-Based Decoding for Multimodal Machine Translation
dc.type Journal article
dc.citation.journalTitle Frontiers in Artificial Intelligence
dc.citation.volumeNumber 4
dc.identifier.digital frai-04-736722
dc.type.dcmi Text
dc.identifier.doihttps://doi.org/10.3389/frai.2021.736722
dc.type.publication publisher version
dc.citation.articleNumber 736722


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record