基于组合神经网络的Sarsa(λ)学习算法

Sarsa (λ) learning algorithm based on stacked neural network

标准的Sarsa(λ)算法对状态空间的要求是离散的且空间较小,而实际问题中很多系统的状态空间是连续的或尽管是离散的但空间较大,这就需要很大的内存来存储状态动作对。为此提出组合神经网络,首先用自组织映射(SOM)神经网络对状态空间进行自适应量化,然后在此基础上用BP网络拟合Q函数。该方法实现了Sarsa(λ)算法在连续和大规模状态空间的泛化。最后,实验结果表明了该方法的有效性。

The standard Sarsa （λ） algorithm requires that the state space is discrete and small. However, in real word it does not satisfy that due to the fact that it may be continuous or discrete but has large state space, so it needs large number of memory to save state-action pairs. Therefore, a stacked neural network is proposed which uses a self-organizing map （SOM） neural network to quantize adaptively the space of states, then a BP network is used to approximate a Q-function. The method realizes the generalization of Sarsa （λ） algorithm in the continuous or large-scale states. The experiment shows the validity of the proposed algorithm.