基于指数移动平均知识蒸馏的神经网络低比特量化方法

Low-Bit Quantization of Neural Network Based on Exponential Moving Average Knowledge Distillation

目前存储和计算成本严重阻碍深度神经网络应用和推广,而神经网络量化是一种有效的压缩方法.神经网络低比特量化存在的显著困难是量化比特数越低,网络分类精度也越低.为了解决这一问题,文中提出基于指数移动平均知识蒸馏的神经网络低比特量化方法.首先利用少量图像进行自适应初始化,训练激活和权重的量化步长,加快量化网络收敛.再引入指数移动平均(EMA)知识蒸馏的思想,利用EMA对蒸馏损失和任务损失进行归一化,指导量化网络训练.在ImageNet、CIFAR-10数据集上的分类任务表明,文中方法可获得接近或超过全精度网络的性能.

Now the memory and computational cost restrict the popularization of deep neural network application,whereas neural network quantization is an effective compression method.As the number of quantized bits is lower,the classification accuracy of neural networks becomes poorer in low-bit quantization of neural networks.To solve this problem,a low-bit quantization method of neural networks based on knowledge distillation is proposed.Firstly,a few images are exploited for adaptive initialization to train the quantization step of activation and weight to speed up the convergence of the quantization network.Then,the idea of exponential moving average knowledge distillation is introduced to normalize distillation loss and task loss and guide the training of quantization network.Experiments on ImageNet and CIFAR-10 datasets show that the performance of the proposed method is close to or better than that of the full precision network.