一种新型激活函数:提高深层神经网络建模能力

Application of Activation Function of Deep Neural Network

修正线性单元做为深层神经网络的激活函数,常被用来处理复杂的函数来提高深层神经网络的建模能力。针对传统修正线性单元提出一种新的激活函数——Tanh ReLU,Tanh ReLU在修正线性单元的基础上为其添加一个负值和边界值,同时保证Tanh ReLU函数在原点处相切,以此克服由于修正线性单元激活函数非零均值激活、极大输出值和原点处不连续的缺点而损害网络的生成。将此新的激活函数应用于MNIST手写数据分类实验以验证其建立的深层神经网络的性能;同时针对网络中不同的超参数的选择,来进一步验证Tanh ReLU对于提高深层神经网络模型性能的影响。实验结果表明:与修正线性单元相比,Tanh ReLU建立的深层神经网络得到了更好的分类结果,实现了提高深层神经网络分类性能的目的。

Rectified linear unit( ReLU) is used as an activation function of deep neural network( DNN),which is often used to deal with complex functions to improve the modeling ability of DNN. A new activation function-Tanh ReLU, is presented for correction of traditional ReLU. Tanh ReLU adds a negative value and boundary value to the ReLU, and guarantees that the Tanh ReLU function is tangent at the origin, thereby overcoming the disadvantages of non-zero mean activation, the maximum output value and discontinuity at the origin of the ReLU that harming the network's training. The new activation function is applied to MNIST handwritten data classification experiments to verify the performance of DNN. Meanwhile, the effect of Tanh ReLU on improving the performance of DNN models is verifed by selecting different hyper parameters of network. These experimental results show that the DNN built by Tanh ReLU gets better classification results, and achieves the purpose of improving the classification performance of DNN.