Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machines

We compare and contrast the statistical physics and quantum physics inspired approaches for unsupervised generative modeling of classical data. The two approaches represent probabilities of observed data using energy-based models and quantum states, respectively. Classical and quantum information pa...

Full description

Bibliographic Details
Main Authors: Song Cheng, Jing Chen, Lei Wang
Format: Article
Language:English
Published: MDPI AG 2018-08-01
Series:Entropy
Subjects:
Online Access:http://www.mdpi.com/1099-4300/20/8/583
id doaj-c968c0ee2cf04b73a3d96aac90272b7c
record_format Article
spelling doaj-c968c0ee2cf04b73a3d96aac90272b7c2020-11-24T23:23:49ZengMDPI AGEntropy1099-43002018-08-0120858310.3390/e20080583e20080583Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born MachinesSong Cheng0Jing Chen1Lei Wang2Institute of Physics, Chinese Academy of Sciences, Beijing 100190, ChinaInstitute of Physics, Chinese Academy of Sciences, Beijing 100190, ChinaInstitute of Physics, Chinese Academy of Sciences, Beijing 100190, ChinaWe compare and contrast the statistical physics and quantum physics inspired approaches for unsupervised generative modeling of classical data. The two approaches represent probabilities of observed data using energy-based models and quantum states, respectively. Classical and quantum information patterns of the target datasets therefore provide principled guidelines for structural design and learning in these two approaches. Taking the Restricted Boltzmann Machines (RBM) as an example, we analyze the information theoretical bounds of the two approaches. We also estimate the classical mutual information of the standard MNIST datasets and the quantum Rényi entropy of corresponding Matrix Product States (MPS) representations. Both information measures are much smaller compared to their theoretical upper bound and exhibit similar patterns, which imply a common inductive bias of low information complexity. By comparing the performance of RBM with various architectures on the standard MNIST datasets, we found that the RBM with local sparse connection exhibit high learning efficiency, which supports the application of tensor network states in machine learning problems.http://www.mdpi.com/1099-4300/20/8/583born machinetensor networkmutual information
collection DOAJ
language English
format Article
sources DOAJ
author Song Cheng
Jing Chen
Lei Wang
spellingShingle Song Cheng
Jing Chen
Lei Wang
Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machines
Entropy
born machine
tensor network
mutual information
author_facet Song Cheng
Jing Chen
Lei Wang
author_sort Song Cheng
title Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machines
title_short Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machines
title_full Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machines
title_fullStr Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machines
title_full_unstemmed Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machines
title_sort information perspective to probabilistic modeling: boltzmann machines versus born machines
publisher MDPI AG
series Entropy
issn 1099-4300
publishDate 2018-08-01
description We compare and contrast the statistical physics and quantum physics inspired approaches for unsupervised generative modeling of classical data. The two approaches represent probabilities of observed data using energy-based models and quantum states, respectively. Classical and quantum information patterns of the target datasets therefore provide principled guidelines for structural design and learning in these two approaches. Taking the Restricted Boltzmann Machines (RBM) as an example, we analyze the information theoretical bounds of the two approaches. We also estimate the classical mutual information of the standard MNIST datasets and the quantum Rényi entropy of corresponding Matrix Product States (MPS) representations. Both information measures are much smaller compared to their theoretical upper bound and exhibit similar patterns, which imply a common inductive bias of low information complexity. By comparing the performance of RBM with various architectures on the standard MNIST datasets, we found that the RBM with local sparse connection exhibit high learning efficiency, which supports the application of tensor network states in machine learning problems.
topic born machine
tensor network
mutual information
url http://www.mdpi.com/1099-4300/20/8/583
work_keys_str_mv AT songcheng informationperspectivetoprobabilisticmodelingboltzmannmachinesversusbornmachines
AT jingchen informationperspectivetoprobabilisticmodelingboltzmannmachinesversusbornmachines
AT leiwang informationperspectivetoprobabilisticmodelingboltzmannmachinesversusbornmachines
_version_ 1725563494560956416