动态环境下数据驱动Q-学习算法

Data-Driven Q-Learning in Dynamic Environment

针对动态环境下强化学习对未知动作的探索和已知最优动作的利用之间难以平衡的问题，提出了一种数据驱动Q-学习算法．该算法首先构建智能体的行为信息系统，通过行为信息系统知识的不确定性建立环境触发机制；依据跟踪环境变化的动态信息，触发机制自适应控制对新环境的探索，使算法对未知动作的探索和已知最优动作的利用达到平衡．用于动态环境下迷宫导航问题的仿真结果表明，该算法达到目标的平均步长比Q-学习算法、模拟退火Q-学习算法和基于探测刷新Q-学习算法缩短了7．79％～84．7％．

It is difficult for reinforcement learning to balance between the exploration of untested actions and the exploitation of known optimum actions in dynamic environment. To address this problem, a data-driven Q-learning algorithm was proposed. In this algorithm, the information system of behavior is constructed for each agent. Then the trigger mechanism of environment is build by the uncertainty of knowledge in the information system of behavior to trace the environmental change. The dynamic information of the environment is used to exploit new environment by the trigger mechanism to achieve the balance between the exploration of untested actions and the exploitation of know optimum actions. The proposed algorithm was applied to grid-world navigation tasks. The simulation results show that compared with the Q-learning, simulated annealing Q-learning （SAQ） and recency-based exploration （RBE） Q-learning algorithms, the proposed algorithm has a high learning efficiency.