Improving the Accuracy of Pruned Network Using Knowledge Distillation

• Main Author: SETYA WIDYAWAN PRAKOSA
• Other Authors: Jenq-Shiou Leu
• Format: Others
• Language: en_US
• Published: 2018
• Online Access: http://ndltd.ncl.edu.tw/handle/u2y997

碩士 === 國立臺灣科技大學 === 電子工程系 === 106 === The introduction of Convolutional Neural Networks (CNN) in image processing field has attracted researchers to explore the applications of CNN itself. Some network designs have been proposed to reach the state of the art capability. However, the current design of neural network remains an issue related to the size of the model. Thus, some researchers introduce to reduce or compress the model size. The compression technique might affect the accuracy of the compressed model compared to the original one. In addition, it may influence the performance of the new model. Furthermore, we need to exploit a new scheme to enhance the accuracy of compressed network. In this study, we explore that Knowledge Distillation (KD) can be integrated to one of pruning methodologies namely pruning filters, as the compression technique, to enhance the accuracy of pruned model. From all experimental results, we conclude that incorporating KD to create a MobileNets model can enhance the accuracy of pruned network without elongating the inference time. We measured the inference time of model trained with KD is just 0.1s longer than that of without KD. Furthermore, by reducing 26.08% of the model size, the accuracy without KD is 63.65% and by incorporating KD, we can enhance to 65.37%. By reducing the size of model using pruning filters, we can deduct the size while the original size of MobileNets is 14.4 MB and reducing 26.08% can decrease the size to 11.3 MB. We also save 0.1 s inference time by compressing the size of model.