基于通道性能度量的神经网络结构搜索算法

Differentiable neural architecture search with channel performance measurement

卷积神经网络的表征与预测能力往往依赖结构合理性,但其主流结构均由人工设计,存在设计难度高、算力要求强、时间开销大等问题.如何让神经网络自主搜索合理结构并节约计算资源是当前的研究重点.目前,基于部分通道连接的可微分结构搜索算法,以其高效的显存利用率在搜索速度和分类性能上表现良好.然而,其针对通道的随机采样策略易造成重要信息丢失,当通道连接不足时性能明显下降.为此,提出一种基于通道性能度量的神经网络结构搜索算法,利用注意力机制提取通道重要性系数,并以此对通道进行排序采样.此外,考虑到预热阶段导致搜索不充分,产生较大离散化误差,在结构权重连续化的过程中设计温度正则化系数,提升权重差异.实验表明,所提算法能够在节约计算资源的基础上搜索出更优的卷积神经网络结构.

The design of convolutional neural network architecture is critical for achieving high characterization and prediction performance.However,manual design can be time-consuming and computationally expensive.Automatic architecture search methods are thus becoming popular.Presently,partial channel connections for memory-efficient differentiable architecture search being efficient in terms of search speed,memory utilization,and classification performance.However,the random sampling strategy can cause the loss of important information.To address these issues,this paper proposes differentiable neural architecture search with channel performance measurement,which extracts importance coefficients of channels with attention mechanism and ranks the channels accordingly to select the most important ones.The proposed algorithm also includes a temperature regularization coefficient to improve the transformation from discrete to continuous architecture weight and reduce discretization errors.Experimental results show that the proposed algorithm achieves good performance while significantly reducing computational resources.