基于掩膜自动编码器的对抗对比蒸馏算法

An Adversarial Contrastive Distillation Algorithm Based on Masked Auto-Encoder

随着人工智能的不断发展,神经网络对不同领域的任务都表现出了优异的性能.然而,对抗样本的存在对神经网络在安全相关领域中的应用提出了挑战.为了改善对抗训练耗时和对抗样本缺乏多样性的问题,本文提出一种使用改进掩膜自动编码器训练教师网络的对比蒸馏算法抵御对抗攻击.首先,为了减弱教师模型对图像全局特征的依赖,教师模型在改进的掩膜自动编码器中学习如何根据可见子块推理遮挡子块的特征.然后,为了减弱对抗干扰的影响,本文采用知识蒸馏和对比学习的方法提升目标模型的对抗鲁棒性,通过知识蒸馏转移教师模型的特征到学生模型减少模型对全局特征的依赖,通过对比学习提升学生模型对图像之间细节特征的识别能力.最后,本文采用标签信息对分类头进行调节确保识别准确率.在ResNet50和WideResNet50中进行的实验表明,CIFAR-10中对抗准确率平均提升11.50%;CIFAR-100中对抗准确率平均提升6.35%.实验结果证明基于掩膜自动编码器的对比蒸馏算法能够通过只生成一次对抗样本减弱对抗干扰的影响,并通过随机掩膜构建多样本视角提升样本多样性,增强神经网络对抗鲁棒性.

With the continuous development of artificial intelligence,neural networks have exhibited exceptional performance across various domains.However,the existence of adversarial samples poses a significant challenge to the application of neural networks in security-related fields.As research progresses,there is an increasing focus on the robustness of neural networks and their inherent performance.This paper aims to improve neural networks to enhance their adversarial robustness.Although adversarial training has shown great potential in improving adversarial robustness,it suffers from the drawback of long running times.This is primarily because it requires generating adversarial samples for the target model at each iteration step.To address the issues of time-consuming adversarial sample generation and lack of diversity in adversarial training,this paper proposes a contrastive distillation algorithm based on masked autoencoders(MAE) to enhance the adversarial robustness of neural networks.Due to the low information density in images,the loss of image pixels caused by masking can often be recovered using neural networks.Thus,masking-based methods are commonly employed to increase sample diversity and improve the feature learning capabilities of neural networks.Given that adversarial training methods often require considerable time to generate adversarial samples,this paper adopts masking methods to mitigate the time-consuming issue of continuously generating adversarial samples during adversarial training.Additionally,randomly occluding parts of the image can effectively enhance sample diversity,which helps create multi-view samples to address the problem of feature singularity in contrastive learning.Firstly,to reduce the teacher model's reliance on global image features,the teacher model learns in an improved masked autoencoder how to infer the features of obscured blocks based on visible sub-blocks.This method allows the teacher model to focus on learning how to reconstruct global features from limited visible parts,thereby enhancing its deep feature learning ability.Then,to mitigate the impact of adversarial interference,this paper employs knowledge distillation and contrastive learning methods to enhance the target model's adversarial robustness.Knowledge distillation reduces the target model's dependence on global features by transferring the knowledge from the teacher model,while contrastive learning enhances the model's ability to recognize fine-grained information among images by leveraging the diversity of the generated multi-view samples.Finally,label information is utilized to adjust the classification head to ensure recognition accuracy.By fine-tuning the classification head with label information,the model can maintain high accuracy in recognizing clean samples while improving its robustness against adversarial attacks.Experimental results conducted on ResNet50 and WideResNet50 demonstrate an average improvement of 11.50% in adversarial accuracy on CIFAR-10 and an average improvement of 6.35% on CIFAR-100.These results validate the effectiveness of the proposed contrastive distillation algorithm based on masked autoencoders.The algorithm attenuates the impact of adversarial interference by generating adversarial samples only once,enhances sample diversity through random masking,and improves the neural network's adversarial robustness.