基于阿当姆斯捷径连接的深度神经网络模型压缩方法

A deep neural network model compression method based on Adams shortcut connection

深度神经网络已经在各类计算机视觉任务中取得了很大的成功,可网络结构设计仍缺乏指导性的准则。大量的理论和经验证据表明,神经网络的深度是它们成功的关键,而深度神经网络的可训练性仍是待解决的问题。将微分方程数值解法——阿当姆斯(Adams)法用于深度神经网络的权重学习,提出一种基于阿当姆斯法的捷径连接(shortcut connection)方式,可提高网络后期的学习精度,压缩模型的规模,使模型变得更有效。尤其对网络层数较少的深度神经网络的可训练性优化效果更明显。以经典的ResNet为例,比较了使用基于Adams法的捷径连接方式的Adams-ResNet与源模型在Cifar10上的性能表现,所提方法在提高识别正确率的同时将源模型的参数量压缩至一半。

Deep neural network has made a great breakthrough in all kinds of computer vision tasks,but there is still a lack of guiding principles in network structure design.A great deal of theoretical and empirical evidence shows that the depth of neural networks is the key to their success,but the trainability of neural networks remains to be solved.In this paper,the numerical method of differential equations(Adams)is used in the weight learning of deep neural network,and a shortcut connection based on Adams method is proposed to improve the learning accuracy of late network,compress the size of model,and make the model more effective.In particular,the trainability optimization effect is obvious for the deep neural network with a small number of layers.Taking the classic ResNet as an example,this paper compares the performance between Adams-ResNet,which uses the shortcut connection based on Adams method,and the source model on Cifar10.The former improves the recognition accuracy while reducing the parameters of the source model to half.