能耗优化的神经网络轻量化方法研究进展

Research Advance on Neural Network Lightweight for Energy Optimization

近年来,神经网络在语音识别、计算机视觉、自然语言处理等领域都取得了良好的进展.大量的神经网络被部署于诸如手机、摄像头等依赖电池或太阳能供电的小型设备.但神经网络参数量大计算复杂,需占用大量计算资源并消耗电能,从而限制了其在资源受限平台上的应用.学术界和工业界逐渐关注于神经网络的高能耗问题.神经网络轻量化方法可以有效地减少参数数量、降低参数精度或优化计算过程从而降低神经网络能耗.本文从能耗优化的角度梳理了神经网络能耗估算方法和神经网络轻量化方法的基本思路,综述了近年来该领域主要研究成果,并提出了能耗估算和能耗优化的神经网络轻量化方法存在的挑战及进一步研究的方向.其中神经网络能耗估算方法包括测量法、分析法和估算法.能耗优化的神经网络轻量化方法包括剪枝、量化、张量分解和知识蒸馏.对于进一步研究方向我们认为,首先需要建立可自适应网络类型的能耗模型;然后需要考虑平衡精度和能耗的轻量化方法.其次需要实现硬件平台可泛化的轻量化方法;最后开发搜索空间可约束的轻量化方法.

Recently neural networks have achieved the great progress in speech recognition,computer vision,natural language processing,and other fields.More and more neural networks are deployed in embedded devices such as mobile phones and cameras,which are relying on batteries or solar energy as their power supply.However,neural networks consume a large amount of storage resources and electric energy,limiting their application on resource-limited platforms.Therefore,both of the academia and industry pay attention to the high energy consumption problem of neural networks.Neural network lightweight method can effectively reduce the number and the precision of parameters and simplify the calculation to optimize energy consumption of neural networks.In this paper,we introduce the basic ideas of neural network energy consumption estimation and neural network lightweight methods for energy optimization,and we summarize the primary research achievements of the field in recent years.Moreover,the challenges and future research trends in both aspects are put forward.The neural network energy consumption estimation methods are categorized as measurement methods,analysis methods and estimation methods.The neural network lightweight methods for energy optimization include pruning,quantization,tensor decomposition and knowledge distillation.We summarize four research directions for the further research.First,an energy consumption model needs to be established to adapt to all neural network schemas.Secondly,the balance between the accuracy and the energy consumption needs to be considered in the neural network lightweight methods.Thirdly,lightweight methods that can be generalized for hardware platforms need to be implemented.Finally,lightweight methods that are able to limit searching space need to be developed.