Robust 2-bit Quantization of Weights in Neural Network Modeled by Laplacian Distribution
Significant efforts are constantly involved in finding manners to decrease the number of bits required for quantization of neural network parameters. Although in addition to compression, in neural networks, the application of quantizer models that are robust to changes in the variance of input dat...
Main Authors: | , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
Stefan cel Mare University of Suceava
2021-08-01
|
Series: | Advances in Electrical and Computer Engineering |
Subjects: | |
Online Access: | http://dx.doi.org/10.4316/AECE.2021.03001 |
id |
doaj-b1abe5127c5d44fbb9f844dce6b67920 |
---|---|
record_format |
Article |
spelling |
doaj-b1abe5127c5d44fbb9f844dce6b679202021-09-04T14:49:14ZengStefan cel Mare University of SuceavaAdvances in Electrical and Computer Engineering1582-74451844-76002021-08-0121331010.4316/AECE.2021.03001Robust 2-bit Quantization of Weights in Neural Network Modeled by Laplacian DistributionPERIC, Z.DENIC, B.DINCIC, M.NIKOLIC, J.Significant efforts are constantly involved in finding manners to decrease the number of bits required for quantization of neural network parameters. Although in addition to compression, in neural networks, the application of quantizer models that are robust to changes in the variance of input data is of great importance, to the best of authors knowledge, this topic has not been sufficiently researched so far. For that reason, in this paper we give preference to logarithmic companding scalar quantizer, which has shown the best robustness in high quality quantization of speech signals, modelled similarly as weights in neural networks, by Laplacian distribution. We explore its performance by performing the exact and asymptotic analysis for low resolution scenario with 2-bit quantization, where we draw firm conclusions about the usability of the exact performance analysis and design of our quantizer. Moreover, we provide a manner to increase the robustness of the quantizer we propose by involving additional adaptation of the key parameter. Theoretical and experimental results obtained by applying our quantizer in processing of neural network weights are very good matched, and, for that reason, we can expect that our proposal will find a way to practical implementation.http://dx.doi.org/10.4316/AECE.2021.03001image classificationneural networksquantizationsignal to noise ratiosource coding |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
PERIC, Z. DENIC, B. DINCIC, M. NIKOLIC, J. |
spellingShingle |
PERIC, Z. DENIC, B. DINCIC, M. NIKOLIC, J. Robust 2-bit Quantization of Weights in Neural Network Modeled by Laplacian Distribution Advances in Electrical and Computer Engineering image classification neural networks quantization signal to noise ratio source coding |
author_facet |
PERIC, Z. DENIC, B. DINCIC, M. NIKOLIC, J. |
author_sort |
PERIC, Z. |
title |
Robust 2-bit Quantization of Weights in Neural Network Modeled by Laplacian Distribution |
title_short |
Robust 2-bit Quantization of Weights in Neural Network Modeled by Laplacian Distribution |
title_full |
Robust 2-bit Quantization of Weights in Neural Network Modeled by Laplacian Distribution |
title_fullStr |
Robust 2-bit Quantization of Weights in Neural Network Modeled by Laplacian Distribution |
title_full_unstemmed |
Robust 2-bit Quantization of Weights in Neural Network Modeled by Laplacian Distribution |
title_sort |
robust 2-bit quantization of weights in neural network modeled by laplacian distribution |
publisher |
Stefan cel Mare University of Suceava |
series |
Advances in Electrical and Computer Engineering |
issn |
1582-7445 1844-7600 |
publishDate |
2021-08-01 |
description |
Significant efforts are constantly involved in finding manners to decrease the number of bits required for
quantization of neural network parameters. Although in addition to compression, in neural networks, the
application of quantizer models that are robust to changes in the variance of input data is of great
importance, to the best of authors knowledge, this topic has not been sufficiently researched so far.
For that reason, in this paper we give preference to logarithmic companding scalar quantizer, which has
shown the best robustness in high quality quantization of speech signals, modelled similarly as weights
in neural networks, by Laplacian distribution. We explore its performance by performing the exact and
asymptotic analysis for low resolution scenario with 2-bit quantization, where we draw firm conclusions
about the usability of the exact performance analysis and design of our quantizer. Moreover, we provide
a manner to increase the robustness of the quantizer we propose by involving additional adaptation of
the key parameter. Theoretical and experimental results obtained by applying our quantizer in processing
of neural network weights are very good matched, and, for that reason, we can expect that our proposal
will find a way to practical implementation. |
topic |
image classification neural networks quantization signal to noise ratio source coding |
url |
http://dx.doi.org/10.4316/AECE.2021.03001 |
work_keys_str_mv |
AT pericz robust2bitquantizationofweightsinneuralnetworkmodeledbylaplaciandistribution AT denicb robust2bitquantizationofweightsinneuralnetworkmodeledbylaplaciandistribution AT dincicm robust2bitquantizationofweightsinneuralnetworkmodeledbylaplaciandistribution AT nikolicj robust2bitquantizationofweightsinneuralnetworkmodeledbylaplaciandistribution |
_version_ |
1717815126328868864 |