Multi-Style Unsupervised Image Synthesis Using Generative Adversarial Nets
Unsupervised cross-domain image-to-image translation is a very active topic in computer vision and graphics. This task has two challenges: 1) lack of paired training data and 2) numerous possible outputs from a single image. The existing methods rely on either paired data or perform one-to-one trans...
Main Authors: | , , |
---|---|
Format: | Article |
Language: | English |
Published: |
IEEE
2021-01-01
|
Series: | IEEE Access |
Subjects: | |
Online Access: | https://ieeexplore.ieee.org/document/9448164/ |
id |
doaj-58302bb3d0a04942abe0819edefc0ef2 |
---|---|
record_format |
Article |
spelling |
doaj-58302bb3d0a04942abe0819edefc0ef22021-06-18T23:00:53ZengIEEEIEEE Access2169-35362021-01-019860258603610.1109/ACCESS.2021.30876659448164Multi-Style Unsupervised Image Synthesis Using Generative Adversarial NetsGuoyun Lv0https://orcid.org/0000-0003-0698-1703Syed Muhammad Israr1https://orcid.org/0000-0002-8306-9905Shengyong Qi2https://orcid.org/0000-0002-3865-2747School of Electronics and Information, Northwestern Polytechnical University, Xi’an, ChinaSchool of Electronics and Information, Northwestern Polytechnical University, Xi’an, ChinaSchool of Electronics and Information, Northwestern Polytechnical University, Xi’an, ChinaUnsupervised cross-domain image-to-image translation is a very active topic in computer vision and graphics. This task has two challenges: 1) lack of paired training data and 2) numerous possible outputs from a single image. The existing methods rely on either paired data or perform one-to-one translation. A novel Multi-Style Unsupervised image synthesis model using Generative Adversarial Nets (MSU-GAN) is proposed in this paper to overcome these disadvantages. Firstly, the encoder-decoder structure is used to map the image to domain-shared content features space and domain-specific style features space. Secondly, to translate an image into another domain, the content code and the style code are combined to synthesize the resulting image. Finally, the bidirectional cycle-consistency loss is used for the unpaired training data; the inter-domain adversarial loss and the reconstruction loss are used to ensure the output image’s realism. Simultaneously, MSU-GAN is able to synthesize multi-style images due to disentangled representation. A Multi-Style Unsupervised Feature-Wise image synthesis model using Generative Adversarial Nets (MSU-FW-GAN) based on the MSU-GAN is proposed for the shape variation tasks. There are two different testing strategies, which include random style transfer and style guide transfer. For objective comparison, the proposed model performs well on all evaluation metrics. The random style transfer experiment results show that compared with CycleGAN on the photo2portraits dataset, MSU-FW-GAN FID, IS scores dropped by 12.77% and 8.06%. For the summer2winter dataset, MSU-GAN FID and IS scores increased by 24.51% and 3.64%. Qualitative results show that without paired training data, MSU-GAN and MSU-FW-GAN can synthesize multi-style and better realistic images on various tasks.https://ieeexplore.ieee.org/document/9448164/Generative Adversarial Netsconvolutional neural networkimage synthesisResNet |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Guoyun Lv Syed Muhammad Israr Shengyong Qi |
spellingShingle |
Guoyun Lv Syed Muhammad Israr Shengyong Qi Multi-Style Unsupervised Image Synthesis Using Generative Adversarial Nets IEEE Access Generative Adversarial Nets convolutional neural network image synthesis ResNet |
author_facet |
Guoyun Lv Syed Muhammad Israr Shengyong Qi |
author_sort |
Guoyun Lv |
title |
Multi-Style Unsupervised Image Synthesis Using Generative Adversarial Nets |
title_short |
Multi-Style Unsupervised Image Synthesis Using Generative Adversarial Nets |
title_full |
Multi-Style Unsupervised Image Synthesis Using Generative Adversarial Nets |
title_fullStr |
Multi-Style Unsupervised Image Synthesis Using Generative Adversarial Nets |
title_full_unstemmed |
Multi-Style Unsupervised Image Synthesis Using Generative Adversarial Nets |
title_sort |
multi-style unsupervised image synthesis using generative adversarial nets |
publisher |
IEEE |
series |
IEEE Access |
issn |
2169-3536 |
publishDate |
2021-01-01 |
description |
Unsupervised cross-domain image-to-image translation is a very active topic in computer vision and graphics. This task has two challenges: 1) lack of paired training data and 2) numerous possible outputs from a single image. The existing methods rely on either paired data or perform one-to-one translation. A novel Multi-Style Unsupervised image synthesis model using Generative Adversarial Nets (MSU-GAN) is proposed in this paper to overcome these disadvantages. Firstly, the encoder-decoder structure is used to map the image to domain-shared content features space and domain-specific style features space. Secondly, to translate an image into another domain, the content code and the style code are combined to synthesize the resulting image. Finally, the bidirectional cycle-consistency loss is used for the unpaired training data; the inter-domain adversarial loss and the reconstruction loss are used to ensure the output image’s realism. Simultaneously, MSU-GAN is able to synthesize multi-style images due to disentangled representation. A Multi-Style Unsupervised Feature-Wise image synthesis model using Generative Adversarial Nets (MSU-FW-GAN) based on the MSU-GAN is proposed for the shape variation tasks. There are two different testing strategies, which include random style transfer and style guide transfer. For objective comparison, the proposed model performs well on all evaluation metrics. The random style transfer experiment results show that compared with CycleGAN on the photo2portraits dataset, MSU-FW-GAN FID, IS scores dropped by 12.77% and 8.06%. For the summer2winter dataset, MSU-GAN FID and IS scores increased by 24.51% and 3.64%. Qualitative results show that without paired training data, MSU-GAN and MSU-FW-GAN can synthesize multi-style and better realistic images on various tasks. |
topic |
Generative Adversarial Nets convolutional neural network image synthesis ResNet |
url |
https://ieeexplore.ieee.org/document/9448164/ |
work_keys_str_mv |
AT guoyunlv multistyleunsupervisedimagesynthesisusinggenerativeadversarialnets AT syedmuhammadisrar multistyleunsupervisedimagesynthesisusinggenerativeadversarialnets AT shengyongqi multistyleunsupervisedimagesynthesisusinggenerativeadversarialnets |
_version_ |
1721372604015575040 |