一种状态自动划分的模糊小脑模型关节控制器值函数拟合方法

Fuzzy cerebellar model arithmetic controller with automatic state partition for value function approximation

在庞大离散状态空间或连续状态空间中,强化学习(RL)需要进行值函数拟合以寻找最优策略.但函数拟合器的结构往往由设计者预先设定,在学习过程中不能动态调整缺乏自适应性.为了自动构建函数拟合器的结构,提出一种可以进行状态自动划分的模糊小脑模型关节控制(FCMAC)值函数拟合方法.该方法利用Bellman误差的变化趋势实现状态自动划分,并且探讨了两种选择划分区域的机制.汽车爬坡问题和机器人足球仿真平台中的实验结果表明新算法能有效拟合值函数,而且利用所提出的函数拟合器智能体可以进行有效的强化学习.

In continuous-state space or large discrete-state space,the reinforcement learning（RL） uses function ap-proximation approaches to represent the value function in seeking the optimal policy.However the structures of function approximators which will greatly influence the learning performance are often designed in advance.To generate the struc-ture of function approximator automatically,a novel function approximator called the automatic state-partition-based fuzzy cerebellar model arithmetic controller（ASP-FCMAC） is proposed.In ASP-FCMAC,the variation tendency of Bellman error is used to determine the best time to perform state partition and two mechanisms are also discussed for determining which state should be partitioned at each time step.Experimental results in solving mountain car problem and RoboCup Keepaway problem demonstrate that ASP-FCMAC can automatically generate the structure of FCMAC for effective rein-forcement learning.