In the past decades, various neural network models have been developed for modeling the behavior of human brain or performing problem-solving through simulating the behavior of human brain. The recurrent neural networ...In the past decades, various neural network models have been developed for modeling the behavior of human brain or performing problem-solving through simulating the behavior of human brain. The recurrent neural networks are the type of neural networks to model or simulate associative memory behavior of human being. A recurrent neural network （RNN） can be generally formalized as a dynamic system associated with two fundamental operators： one is the nonlinear activation operator deduced from the input-output properties of the involved neurons, and the other is the synaptic connections （a matrix） among the neurons. Through carefully examining properties of various activation functions used, we introduce a novel type of monotone operators, the uniformly pseudo-projectionanti-monotone （UPPAM） operators, to unify the various RNN models appeared in the literature. We develop a unified encoding and stability theory for the UPPAM network model when the time is discrete. The established model and theory not only unify but also jointly generalize the most known results of RNNs. The approach has lunched a visible step towards establishment of a unified mathematical theory of recurrent neural networks.展开更多
基金Supported by the National Basic Research Program of China （973 Program） （Grant No. 2007CB311002）, the National Nature Science Foundation of China （Grant No. 61075054） and the Fundamental Research Funds for the Central Universities （Grant No. xjj20100087）
文摘In the past decades, various neural network models have been developed for modeling the behavior of human brain or performing problem-solving through simulating the behavior of human brain. The recurrent neural networks are the type of neural networks to model or simulate associative memory behavior of human being. A recurrent neural network （RNN） can be generally formalized as a dynamic system associated with two fundamental operators： one is the nonlinear activation operator deduced from the input-output properties of the involved neurons, and the other is the synaptic connections （a matrix） among the neurons. Through carefully examining properties of various activation functions used, we introduce a novel type of monotone operators, the uniformly pseudo-projectionanti-monotone （UPPAM） operators, to unify the various RNN models appeared in the literature. We develop a unified encoding and stability theory for the UPPAM network model when the time is discrete. The established model and theory not only unify but also jointly generalize the most known results of RNNs. The approach has lunched a visible step towards establishment of a unified mathematical theory of recurrent neural networks.
