基于元学习的无数据模型压缩

Compression of data-free model based on meta-learning

针对现有深度学习无数据蒸馏框架下,数据合成效率低下以及蒸馏模型性能不足的问题,提出一种基于元学习的快速数据合成方法。通过批量标准化层的平均值和方差,及动量自适应调整,提取可重用特征,对特定任务执行少量更新,达到提高数据合成效率的目的;通过提出同异构教师鉴别器提取双采样知识,解决数据样本多样性与泛化性问题;改进传统知识蒸馏损失,采用学生自恢复蒸馏,通过一个生成块,提高模型性能。实验结果表明,提出方法优于现有无数据蒸馏方法。

Aiming at the low efficiency of data synthesis and the insufficient performance of distillation model under the existing deep learning framework of data-free distillation,a fast data synthesis method based on meta-learning was proposed.To improve the efficiency of data synthesis,The mean and variance of the batch normalization layer were adaptively adjusted with momentum to extract reusable features and perform minor updates for specific tasks.The heterogeneous teacher discriminator was proposed to extract double sampling knowledge to solve the problem of diversity and generalization of data samples.Using student self-recovery distillation,through a generating block,the performance of the model was improved.Experimental results show that the proposed method is superior to the existing distillation method in the absence of data.