摘要
为提高Softplus函数在神经网络中的性能,针对Softplus函数的缺点提出了一种基于改进Softplus激活函数的卷积神经网络模型。根据“输出均值接近零的激活函数可以提升神经网络的学习性能”原理,首先对原函数的输出向下平移缓解Softplus激活函数的均值偏移现象;然后对调整后的输出乘以一个参数调整函数在正数部份的坡度和负数部分的饱和点位置,以缓解在训练过程中出现的梯度消失/溢出现象。最后在MNIST、CIFAR-10和CIFAR-100数据集上的实验结果表明,同其它常用激活函数相比,改进后的Softplus函数取得了较好的识别率。
To solve this problem,this paper proposes a convolutional neural network model based on the improved Softplus activation function.According to the principle that"the activation function whose average output is close to zero can improve the learning performance of the neural network",the output value of the original function is shifted downward to alleviate the offset phenomenon of the Softplus activation function firstly.Then the offset function output is multiplied by a parameter to adjust the slope and the saturation point of the function.This is used to alleviate the gradient disappearance/overflow phenomenon that occurs during training.Finally,the experimental results on the MNIST,CIFAR-10 and CIFAR-100 datasets show that the improved Softplus function achieves a better recognition rate than other commonly used activation functions.
作者
孙克雷
虞佳明
孙刚
SUN Kelei;YU Jiaming;SUN Gang(School of Computer Science and Engineering,Anhui University of Science&Technology,Huainan Anhui 232001,China;School of Computer and Information Engineering,Fuyang Normal University,Fuyang Anhui 236037,China)
出处
《阜阳师范学院学报(自然科学版)》
2020年第1期75-79,共5页
Journal of Fuyang Normal University(Natural Science)
基金
国家自然科学基金项目(61471004)资助。
关键词
卷积神经网络
激活函数
Softplus函数
梯度消失
均值偏移
convolutional neural network
activation function
Softplus function
gradient disappear
offset phenomenon