分块压缩学习剪枝算法

Chunked Compression Learning Algorithm

为压缩网络剪枝过程中较大的搜索空间,从预训练深度神经网络中找到最佳的稀疏网络结构,本文提出一种基于遗传算法和知识蒸馏的分块压缩学习算法(CCLA).首先,在预定义压缩空间中将学生网络初始化为一个稀疏网络结构.然后,将教师网络和学生网络按层划分多个块网络,在教师网络的监督下对学生网络进行逐块压缩.最后,使用遗传算法搜索学生网络中每个块网络的稀疏结构.在CIFAR-10数据集上对VGG-16网络和ResNet-110网络的实验结果表明,本文所提方法性能表现良好,例如,在CIFAR-10数据集上对VGG-16进行压缩,可压缩掉87.82%的参数和70.4%的浮点运算(Floating Point of operations,FLOPs),而精度损失仅为0.37%.

To reduce the large search space in the network pruning process,and find the optimal sparse structure from a baseline network,this paper proposes a chunked compression learning algorithm(CCLA)based on genetic algorithm and knowledge distillation.First,the student network is initialized to a sparse network structure in the predefined compression space.Then,the teacher and student networks are divided into multiple blocks,and block-wise compression of the student network is conducted under the supervision of the teacher network.Finally,a genetic algorithm is used to find sparse structure of each block in student network.Experimental results of pruning VGG-16 and ResNet-110 networks on the CIFAR-10 dataset show that the proposed method performs well.For instance,compressing of VGG-16 on the CIFAR-10 dataset,pruned 87.82%of the parameters and 70.4%of the Floating Point of Operations(FLOPs)with only 0.37%accuracy loss.