Variational Information Bottleneck for Semi-Supervised Classification
In this paper, we consider an information bottleneck (IB) framework for semi-supervised classification with several families of priors on latent space representation. We apply a variational decomposition of mutual information terms of IB. Using this decomposition we perform an analysis of several re...
Main Authors: | , , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
MDPI AG
2020-08-01
|
Series: | Entropy |
Subjects: | |
Online Access: | https://www.mdpi.com/1099-4300/22/9/943 |
id |
doaj-699f1fb9fbd7405789af47065e8c8b4f |
---|---|
record_format |
Article |
spelling |
doaj-699f1fb9fbd7405789af47065e8c8b4f2020-11-25T03:50:09ZengMDPI AGEntropy1099-43002020-08-012294394310.3390/e22090943Variational Information Bottleneck for Semi-Supervised ClassificationSlava Voloshynovskiy0Olga Taran1Mouad Kondah2Taras Holotyak3Danilo Rezende4Department of Computer Science, University of Geneva, 1227 Carouge, SwitzerlandDepartment of Computer Science, University of Geneva, 1227 Carouge, SwitzerlandDepartment of Computer Science, University of Geneva, 1227 Carouge, SwitzerlandDepartment of Computer Science, University of Geneva, 1227 Carouge, SwitzerlandDeepMindIn this paper, we consider an information bottleneck (IB) framework for semi-supervised classification with several families of priors on latent space representation. We apply a variational decomposition of mutual information terms of IB. Using this decomposition we perform an analysis of several regularizers and practically demonstrate an impact of different components of variational model on the classification accuracy. We propose a new formulation of semi-supervised IB with hand crafted and learnable priors and link it to the previous methods such as semi-supervised versions of VAE (M1 + M2), AAE, CatGAN, etc. We show that the resulting model allows better understand the role of various previously proposed regularizers in semi-supervised classification task in the light of IB framework. The proposed IB semi-supervised model with hand-crafted and learnable priors is experimentally validated on MNIST under different amount of labeled data.https://www.mdpi.com/1099-4300/22/9/943information bottleneck principledeep networkssemi-supervised classificationlatent space representationhand crafted priorslearnable priors |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Slava Voloshynovskiy Olga Taran Mouad Kondah Taras Holotyak Danilo Rezende |
spellingShingle |
Slava Voloshynovskiy Olga Taran Mouad Kondah Taras Holotyak Danilo Rezende Variational Information Bottleneck for Semi-Supervised Classification Entropy information bottleneck principle deep networks semi-supervised classification latent space representation hand crafted priors learnable priors |
author_facet |
Slava Voloshynovskiy Olga Taran Mouad Kondah Taras Holotyak Danilo Rezende |
author_sort |
Slava Voloshynovskiy |
title |
Variational Information Bottleneck for Semi-Supervised Classification |
title_short |
Variational Information Bottleneck for Semi-Supervised Classification |
title_full |
Variational Information Bottleneck for Semi-Supervised Classification |
title_fullStr |
Variational Information Bottleneck for Semi-Supervised Classification |
title_full_unstemmed |
Variational Information Bottleneck for Semi-Supervised Classification |
title_sort |
variational information bottleneck for semi-supervised classification |
publisher |
MDPI AG |
series |
Entropy |
issn |
1099-4300 |
publishDate |
2020-08-01 |
description |
In this paper, we consider an information bottleneck (IB) framework for semi-supervised classification with several families of priors on latent space representation. We apply a variational decomposition of mutual information terms of IB. Using this decomposition we perform an analysis of several regularizers and practically demonstrate an impact of different components of variational model on the classification accuracy. We propose a new formulation of semi-supervised IB with hand crafted and learnable priors and link it to the previous methods such as semi-supervised versions of VAE (M1 + M2), AAE, CatGAN, etc. We show that the resulting model allows better understand the role of various previously proposed regularizers in semi-supervised classification task in the light of IB framework. The proposed IB semi-supervised model with hand-crafted and learnable priors is experimentally validated on MNIST under different amount of labeled data. |
topic |
information bottleneck principle deep networks semi-supervised classification latent space representation hand crafted priors learnable priors |
url |
https://www.mdpi.com/1099-4300/22/9/943 |
work_keys_str_mv |
AT slavavoloshynovskiy variationalinformationbottleneckforsemisupervisedclassification AT olgataran variationalinformationbottleneckforsemisupervisedclassification AT mouadkondah variationalinformationbottleneckforsemisupervisedclassification AT tarasholotyak variationalinformationbottleneckforsemisupervisedclassification AT danilorezende variationalinformationbottleneckforsemisupervisedclassification |
_version_ |
1724491953881481216 |