深度学习中的优化算法研究

Research on Optimization Algorithms in Deep Learning

深度学习作为处理神经网络的一个热门研究方向,在近些年来备受关注。深度学习模型是一个多层次的网络结构,评价模型最终效果优劣的网络参数必须通过深度学习优化器进行训练,因此深度学习中的优化算法成为了国内外的研究热点。本文对深度学习中的一阶优化算法进行综述,首先介绍了经典的随机梯度下降及其动量变体优化算法,然后介绍了近年更流行的自适应学习率优化算法,最后对未来深度学习中优化算法的发展进行了总结与展望。

As a popular research direction for processing neural networks, deep learning has attracted much attention in recent years. The deep learning model is a multi-layered network structure, and the network parameters that evaluate the final effect of the model must be trained by the deep learning optimizers, so the optimization algorithms in deep learning have become a research hotspot at home and abroad. In this paper, the first-order optimization algorithms in deep learning are reviewed. Firstly, the classical stochastic gradient descent and its momentum variant optimization algorithms are introduced. Then, the more popular adaptive learning rate optimization algorithms in recent years are introduced. Finally, the development of optimization algorithms in deep learning in the future is summarized and prospected.