基于动量项分离的深度学习优化算法

Deep Learning and Optimization Algorithm Based on Momentum Separation

针对神经网络处理参数更新的优化算法中出现的局部最优点振荡问题,改进带动量项的随机梯度下降算法,提出了一种动量项分离的优化算法。通过计算当前时刻目标函数的曲率半径,根据阈值适时分离动量项,从而缓解局部最优点振荡问题。实验表明,动量项分离的优化算法能够适用于不同的模型结构和不同数据集。相较于带动量项的随机梯度下降算法,具有更高的准确度,能够更快地稳定收敛。与同类一阶动量算法相比,其准确率上升明显,为深度神经网络的参数更新提供了一种新的有效的解决方案。

Aiming at the local optimal oscillation problem in the optimization algorithm of neural network processing parameter update, the stochastic gradient descent algorithm with momentum item was improved, and an optimization algorithm for separation of momentum item was proposed. The momentum items were separated to alleviate the local optimal oscillation based on the threshold value, which was calculated by the radius of curvature of the objective function. Experiments show that the optimization algorithm for the separation of momentum term can be applied to different model structures and different data sets. Compared with the stochastic gradient descent algorithm with momentum, it has higher accuracy and can converge more quickly. Compared with similar first-order momentum algorithms, its accuracy has increased significantly, which provides a new and effective solution for deep neural network parameter update.