修剪算法的信息几何分析

Information Geometric Analysis of Pruning Algorithm

修剪法是确定和优化神经网络结构的重要方法之一.当前对修剪法的研究大多集中在方法描述上,对于修剪法内在机理的研究尚不多见,而研究修剪的内在机理可以为修剪策略提供理论基础和依据.从信息几何的角度研究了修剪法的内在机理,给出了神经网络结构修剪法的信息几何理论解释,利用神经流形参数结构的层次性,将修剪法表述为一系列从当前模型流形到其子流形的信息投影过程,在此基础上提出了新的修剪算法,并给出了算法可行性与有效性的实验验证.

Pruning algorithm is an important method to set up and optimize the structure of neural network model. The research on pruning nowadays mostly focuses on the algorithm description while less effort is spent on its immanent mechanism. Research on its mechanism can provide theoretical basis for pruning strategy. The immanent mechanism of pruning is analyzed based on information geometry and a set of theoretical explanation of pruning is given. The pruning process is depicted as a series of information projections from the current model manifold to its submanifolds utilizing the hierarchical structure of neural manifold parameter space. A new pruning algorithm is presented based on the theoretical analysis and its validity and the efficiency is verified by experiments.