基于中间层频域特征蒸馏的元学习算法

Metalearning algorithm for frequency domain featuredistillation in intermediate layers

针对目前知识蒸馏忽略特征图全局统计信息的问题,提出利用离散余弦变换(DCT)频域释义的全局统计特征知识,并对各类别的频域特征使用Logistic分类器进行二分类,提取类间差异性信息.利用元学习算法(MPL)对教师模型参数进行更新,使教师模型能动态调整所传递的频域特征.实验模型在CIFAR-10,CIFAR-100以及ImageNet 2012数据集上有0.12%~0.16%的精度提升,结果表明,频域特征与类间相似性信息为学生模型的训练提供更多有用的知识,且两模型的知识交互更有利于教师的知识迁移.

Aiming at the problem that knowledge distillation nowadays tends to ignore global statistical information about intermediate layers,the global statistical knowledge of DCT frequency domain interpretation was proposed.The logistic classifier was also utilized for binary classification in terms of frequency domain features for each class,to extract information on inter-class differences.In addition,by referring to parameter update algorithm of teacher model(Meta Pseudo Labels,MPL),the as-proposed models can dynamically adjust the frequency domain features.The experimental models in CIFAR-10,CIFAR-100 and ImageNet 2012 data sets witness improvements of their accuracy rates ranging from 0.12%to 0.16%.The results demonstrate that frequency domain features and inter-class differences can make the training of student models more instructive while making knowledge interaction between both models more favorable to knowledge transfer by teachers.