基于ArcReLU函数的神经网络激活函数优化研究

Optimization of Activation Function in Neural Network Based on ArcReLU Function

近年来深度学习发展迅猛。由于深度学习的概念源于神经网络,而激活函数更是神经网络模型在学习理解非线性函数时不可或缺的部分,因此本文对常用的激活函数进行了研究比较。针对常用的激活函数在反向传播神经网络中具有收敛速度较慢、存在局部极小或梯度消失的问题,将Sigmoid系和ReLU系激活函数进行了对比,分别讨论了其性能,详细分析了几类常用激活函数的优点及不足,并通过研究Arctan函数在神经网络中应用的可能性,结合ReLU函数,提出了一种新型的激活函数ArcReLU。实验证明,该函数既能显著加快反向传播神经网络的训练速度,又能有效降低训练误差并避免梯度消失的问题。

Deep learning has developed rapidly in recent years.The concept of deep learning originates from the neural networks.And the activation function is an indispensable part of the neural network model in learning to understand non-linear functions.Therefore,the common activation functions are studied and compared,aiming at the problems of slow convergence speed,local minimum or gradient disappearance of the commonly used activation functions in back propagation neural networks.In this paper,the Sigmoid and ReLU activation functions are compared,their performances are discussed respectively,and the advantages and disadvantages of several common activation functions are analyzed in detail.Finally,a new activation function,ArcReLU,is proposed by studying the possibility of applying Arctan functions in neural networks and combining with ReLU functions.Experiments show that the function can not only significantly accelerate the training speed of BP neural network,but also effectively reduce the training error and avoid the problem of gradient disappearance.