深度神经网络知识蒸馏综述

A Review of Knowledge Distillation in Deep Neural Networks

深度神经网络在计算机视觉、自然语言处理、语音识别等多个领域取得了巨大成功,但是随着网络结构的复杂化,神经网络模型需要消耗大量的计算资源和存储空间,严重制约了深度神经网络在资源有限的应用环境和实时在线处理的应用上的发展。因此,需要在尽量不损失模型性能的前提下,对深度神经网络进行压缩。本文介绍了基于知识蒸馏的神经网络模型压缩方法,对深度神经网络知识蒸馏领域的相关代表性工作进行了详细的梳理与总结,并对知识蒸馏未来发展趋势进行展望。

Deep neural networks have achieved great success in computer vision, natural language processing, speech recognition and other fields. However, with the complexity of network structure, the neural network model needs to consume a lot of computing resources and storage space, which seriously restricts the development of deep neural network in the resource limited application environment and real-time online processing application. Therefore, it is necessary to compress the deep neural network without losing the performance of the model as much as possible. This article introduces the neural network model compression method based on knowledge distillation, combs and summarizes the relevant representative works in the field of deep neural network knowledge distillation in detail, and prospects the future development trend of knowledge distillation.