摘要
张量训练(TT)分解和Tucker分解是两种有效的卷积神经网络压缩方法。然而,TT和Tucker分解分别面临空间结构信息丢失与计算复杂度高等问题。为解决上述问题,文中考虑了网络结构的信息保留率和资源占用情况,采用学习-压缩(LC)算法的约束型压缩框架,提出了一种基于TT-Tucker分解的无预训练LC卷积神经网络压缩方法(TTLC)。TT-LC方法包括学习步骤和压缩步骤两个部分。学习步骤不需要预训练过程,采用了指数循环学习率方法以提高训练准确率。而在压缩步骤,文中根据TT和Tucker分解的优点以及贝叶斯规则选取全局最优秩的特性,运用经验变分贝叶斯矩阵分解(EVBMF)和贝叶斯优化(BayesOpt)选出合理的秩以指导张量分解,采用TT-LC方法压缩训练后的模型。TT-LC方法既降低了空间结构信息丢失率和计算复杂度,又解决了张量的秩选取不合理导致模型准确率显著下降的问题,可实现模型的双重贝叶斯选秩和双重压缩,获得最优的压缩模型。最后,采用ResNets和VGG网络在CIFAR10与CIFAR100数据集上进行实验。结果表明:对于ResNet32网络,相比于基准方法,文中方法在准确率为92.22%的情况下,获得了69.6%的参数量压缩率和66.7%的浮点计算量压缩率。
Tensor training(TT)decomposition and Tucker decomposition are two effective compression methods for convolutional neural networks.However,TT and Tucker decomposition face the problems of spatial structure information loss and high computational complexity respectively.To solve the above problems,this paper considered the information retention rate and resource occupancy of the network structure and proposed a LC convolutional neural network compressed method(TT-LC)without pre-training based on TT-Tucker decomposition,adopting the learning-compression(LC)algorithm constraint compression framework.The TT-LC method includes two parts:learning step and compression step.The learning step didn’t not need the pre-training process,and adopted the exponential cyclic learning rate method to improve the training accuracy.In the compression step,this paper selected the global optimal rank according to the advantages of TT and Tucker decomposition and the characteristics of Bayes rule,and used empirically variable Bayesian matrix factorization(EVBMF)and Bayesian optimization(BayesOpt)to select reasonable ranks to guide tensor decomposition.The TT-LC method was used to compress the trained model.TT-LC method not only reduces the loss rate of spatial structure information and computational complexity,but also solves the problem that the unreasonable rank selection of the tensor leads to the significant decrease in model accuracy.It can realize the double Bayesian rank selection and double compression of the model,and obtains the optimal compression model.Finally,experiments were carried out on CIFAR10 and CIFAR100 datasets using ResNets and VGG networks.The results show that for ResNet32 network,compared with the benchmark method,the proposed method achieved a compression rate of parameter quantity of 69.6%and a floating point computation compression rate of 66.7%with the accuracy of 92.22%.
作者
刘微容
张志强
张宁
孟家豪
张敏
刘婕
LIU Weirong;ZHANG Zhiqiang;ZHANG Ning;MENG Jiahao;ZHANG Min;LIU Jie(College of Electrical and Information Engineering,Lanzhou University of Technology,Lanzhou 730050,Gansu,China)
出处
《华南理工大学学报(自然科学版)》
EI
CAS
CSCD
北大核心
2024年第7期29-38,共10页
Journal of South China University of Technology(Natural Science Edition)
基金
国家自然科学基金资助项目(62261032)
甘肃省自然科学基金资助项目(22JR5RA272)
甘肃省重点人才项目。
关键词
卷积神经网络
网络压缩
张量分解
贝叶斯优化
约束型压缩
convolutional neural network
network compression
tensor decomposition
Bayesian optimization
constrained compression