多层感知器模型互反奇异性区域学习动态的理论分析

Theoretical analysis of learning dynamics near the opposite singularities in multilayer perceptrons

多层感知器神经网络(MLPs)的学习过程经常发生一些奇异性行为,容易陷入平坦区,这都和MLPs的参数空间中存在的奇异性区域有直接关系.当MLPs的两个隐节点的权值接近互反时,置换对称性会导致学习困难.对MLPs的互反奇异性区域附近的学习动态进行分析.本文首先得到了平均学习方程的解析表达式,然后给出了互反奇异性区域附近的理论学习轨迹,并通过数值方法得到了其附近的实际学习轨迹.通过仿真实验,分别观察了MLPs的平均学习动态,批处理学习动态和在线学习动态,并进行了比较分析.

Owing to the existence of singularities in the parameter space,multilayer perceptrons （MLPs） may behave extremely slowly in learning or even be trapped in plateaus.When weights of two hidden units are nearly mutually opposite,the learning process will encounter difficulties because of the permutation symmetry.We investigate the learning dynamics of MLPs near opposite singularities,and derive the analytical expressions for averaged learning equations.Then,we obtain the theoretical learning trajectories near the opposite singularities.Furthermore,real learning trajectories near the opposite singularities are also calculated by using numerical methods.In simulations,we study the averaged learning dynamics,the batch mode learning dynamics and the online learning dynamics,respectively.