用于神经网络权值稀疏化的L_(1/2)正则化方法

L_(1/2) regularization methods for weights sparsification of neural networks

在保证适当学习精度前提下,神经网络的神经元个数应该尽可能少(结构稀疏化),从而降低成本,提高稳健性和推广精度.本文采用正则化方法研究前馈神经网络的结构稀疏化.除了传统的用于稀疏化的L1正则化之外,本文主要采用近几年流行的L1/2正则化.为了解决L1/2正则化算子不光滑、容易导致迭代过程振荡这一问题,本文试图在不光滑点的一个小邻域内采用磨光技巧,构造一种光滑化L1/2正则化算子,希望达到比L1正则化更高的稀疏化效率.本文综述了近年来作者在用于神经网络稀疏化的L1/2正则化的一些工作,涉及的神经网络包括BP前馈神经网络、高阶神经网络、双并行前馈神经网络,以及Takagi-Sugeno模糊模型.

On the premise of appropriate learning accuracy, the number of the neurons of a neural network should be as less as possible （constructional sparsification）, so as to reduce the cost, and to improve the robustness and the generalization accuracy. We study the constructional sparsification of feedforward neural networks by using regularization methods. Apart from the traditional L1/2 regularization for sparsification, we mainly use the L1/2 regularization. To remove the oscillation in the iteration process due to the nonsmoothness of the L1/2 regularizer, we propose to smooth it in a neighborhood of the nonsmooth point to get a smoothing L1/2 regularizer. By doing so, we expect to improve the efficiency of the L1/2 regularizer so as to surpass the L1 regularizer. Some of our recent works in this respect are summarized in this paper, including the works on BP feedforward neural networks, higher order neural networks, double parallel neural networks and Takagi-Sugeno fuzzy models.