深度学习经验损失函数地貌分析研究进展

Empirical loss landscape analysis in deep learning:A survey

经验损失函数地貌分析是揭示深度学习优化问题易优性本质的有效途径,已成为机器学习及数学优化领域的重要研究方向.本文回顾了经验损失函数地貌分析的研究进展和挑战,主要包括局部极小点的数量与空间分布、最优点之间的连通性、临界点的最优性、梯度下降法收敛性、经验损失函数地貌可视化等.

Empirical loss landscape analysis is critical to reveal reasons why deep networks are easily optimizable,and has aroused considerable interests in machine learning and mathematical optimization.The main goal of this investigation is to provide a comprehensive state-of-the-art review of the empirical loss landscape analysis,including number and spatial distribution of local minima,connectivity between global optima,global optimality of critical points,convergence of gradient descent,and visualization of empirical loss landscape.This review also identifies challenges and highlights opportunities for future work.