基于气味奖励引导的Q-learning环境认知方法

Q-learning environment recognition method based on odor-reward shaping

Q-learning作为一种无模型的值迭代强化学习算法,被广泛应用于移动机器人在非结构环境下的导航任务中。针对Q-learning在移动机器人导航中环境探索和利用存在矛盾关系导致收敛速度慢的问题,该文在Q-learning算法的基础上,受啮齿类动物可以利用嗅觉线索来进行空间定向和导航的启发,提出一种基于气味奖励引导的Q-learning环境认知策略。该算法通过改善Q-learning中的动作选择策略来减少对环境的无用探索,在动作选择策略中融入了环境气味奖励的引导,并提出了嗅觉因子来平衡动作选择策略中Q-learning和气味奖励引导的权重关系。为了验证算法的有效性,在Tolman老鼠实验所用的迷宫环境中进行了仿真实验,动态仿真结果表明,相比Q-learning算法,基于气味奖励引导的Q-learning算法在环境认知过程中,可减少对环境的无用探索,并增强对环境的认知学习能力,且提高算法的收敛速度。

Q-learning is a model-free iterative reinforcement learning algorithm that is widely used for navigating mobile robots in unstructured environments. However, the exploration and utilization of the environmental data limits the Q-learning convergence speed for mobile robot navigation. This study used the Q-learning algorithm and the fact that rodents use olfactory cues for spatial orientation and navigation to develop a Q-learning environmental cognitive strategy based on odor-reward shaping. This algorithm reduces useless exploration of the environment by improving the Q-learning action selection strategy. Environmental odor information is integrated into the algorithm with the olfactory factor used to weight the Q-learning and the odor-reward shaping in the action selection strategy. The algorithm effectiveness is evaluated in a simulation of movement in the labyrinth environment used in the Tolman mouse experiment. The results show that the Q-learning algorithm with odor-reward shaping reduces useless exploration of the environment, enhances cognitive learning of the environment, and improves the algorithm convergence speed.