Compressing by Learning in a Low-Rank and Sparse Decomposition Form
Low-rankness and sparsity are often used to guide the compression of convolutional neural networks (CNNs) separately. Since they capture global and local structure of a matrix respectively, we combine these two complementary properties together to pursue better network compression performance. Most...
Main Authors: | , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
IEEE
2019-01-01
|
Series: | IEEE Access |
Subjects: | |
Online Access: | https://ieeexplore.ieee.org/document/8871134/ |
id |
doaj-532727f047cb4ec09ac2f32c3a0a7aa3 |
---|---|
record_format |
Article |
spelling |
doaj-532727f047cb4ec09ac2f32c3a0a7aa32021-03-29T23:20:24ZengIEEEIEEE Access2169-35362019-01-01715082315083210.1109/ACCESS.2019.29478468871134Compressing by Learning in a Low-Rank and Sparse Decomposition FormKailing Guo0https://orcid.org/0000-0003-4753-9022Xiaona Xie1Xiangmin Xu2Xiaofen Xing3School of Electronic and Information Engineering, South China University of Technology, Guangzhou, ChinaSchool of Electronic and Information Engineering, South China University of Technology, Guangzhou, ChinaSchool of Electronic and Information Engineering, South China University of Technology, Guangzhou, ChinaSchool of Electronic and Information Engineering, South China University of Technology, Guangzhou, ChinaLow-rankness and sparsity are often used to guide the compression of convolutional neural networks (CNNs) separately. Since they capture global and local structure of a matrix respectively, we combine these two complementary properties together to pursue better network compression performance. Most existing low-rank or sparse compression methods compress the networks by approximating pre-trained models. However, the optimal solutions to pre-trained models may not be optimal to compressed networks with low-rank or sparse constraints. In this paper, we propose a low-rank and sparse learning framework that trains the compressed network from scratch. Our compressing process can be described as the following three stages. (a) In the structure designing stage, we decompose a weight matrix into sum of low-rank matrix and sparse matrix, and then the low-rank matrix is further factorized into product of two small matrices. (b) In training stage, we add ℓ<sub>1</sub> regularization to the loss function to force the sparse matrix to be sparse. (c) In the post-processing stage, we remove the unimportant connection of sparse matrix according to its energy distribution. The pruning process in the post-processing stage reserves most of capacity of the network and keeps the performance of the network to a great extent. The performance can be further improved with fine-tuning, along with sparse masked convolution. Experiments on several common datasets demonstrate our model is superior to other network compression methods based on low-rankness or sparsity. On CIFAR-10, our method compresses VGGNet-19 to 3.14% and PreActResNet-56 to 29.78% without accuracy drop. 62.43% of parameters of ResNet-50 are reduced with 0.55% top-5 accuracy loss on ImageNet.https://ieeexplore.ieee.org/document/8871134/Convolutional neural networkslow-ranksparsenetwork compression |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Kailing Guo Xiaona Xie Xiangmin Xu Xiaofen Xing |
spellingShingle |
Kailing Guo Xiaona Xie Xiangmin Xu Xiaofen Xing Compressing by Learning in a Low-Rank and Sparse Decomposition Form IEEE Access Convolutional neural networks low-rank sparse network compression |
author_facet |
Kailing Guo Xiaona Xie Xiangmin Xu Xiaofen Xing |
author_sort |
Kailing Guo |
title |
Compressing by Learning in a Low-Rank and Sparse Decomposition Form |
title_short |
Compressing by Learning in a Low-Rank and Sparse Decomposition Form |
title_full |
Compressing by Learning in a Low-Rank and Sparse Decomposition Form |
title_fullStr |
Compressing by Learning in a Low-Rank and Sparse Decomposition Form |
title_full_unstemmed |
Compressing by Learning in a Low-Rank and Sparse Decomposition Form |
title_sort |
compressing by learning in a low-rank and sparse decomposition form |
publisher |
IEEE |
series |
IEEE Access |
issn |
2169-3536 |
publishDate |
2019-01-01 |
description |
Low-rankness and sparsity are often used to guide the compression of convolutional neural networks (CNNs) separately. Since they capture global and local structure of a matrix respectively, we combine these two complementary properties together to pursue better network compression performance. Most existing low-rank or sparse compression methods compress the networks by approximating pre-trained models. However, the optimal solutions to pre-trained models may not be optimal to compressed networks with low-rank or sparse constraints. In this paper, we propose a low-rank and sparse learning framework that trains the compressed network from scratch. Our compressing process can be described as the following three stages. (a) In the structure designing stage, we decompose a weight matrix into sum of low-rank matrix and sparse matrix, and then the low-rank matrix is further factorized into product of two small matrices. (b) In training stage, we add ℓ<sub>1</sub> regularization to the loss function to force the sparse matrix to be sparse. (c) In the post-processing stage, we remove the unimportant connection of sparse matrix according to its energy distribution. The pruning process in the post-processing stage reserves most of capacity of the network and keeps the performance of the network to a great extent. The performance can be further improved with fine-tuning, along with sparse masked convolution. Experiments on several common datasets demonstrate our model is superior to other network compression methods based on low-rankness or sparsity. On CIFAR-10, our method compresses VGGNet-19 to 3.14% and PreActResNet-56 to 29.78% without accuracy drop. 62.43% of parameters of ResNet-50 are reduced with 0.55% top-5 accuracy loss on ImageNet. |
topic |
Convolutional neural networks low-rank sparse network compression |
url |
https://ieeexplore.ieee.org/document/8871134/ |
work_keys_str_mv |
AT kailingguo compressingbylearninginalowrankandsparsedecompositionform AT xiaonaxie compressingbylearninginalowrankandsparsedecompositionform AT xiangminxu compressingbylearninginalowrankandsparsedecompositionform AT xiaofenxing compressingbylearninginalowrankandsparsedecompositionform |
_version_ |
1724189735568539648 |