模糊Q学习的足球机器人双层协作模型

A double-layer decision-making model based on fuzzy Q-learning for robot soccer

针对传统的足球机器人3层决策模型存在决策不连贯的问题和缺乏适应性与学习能力的缺点,提出了一种基于模糊Q学习的足球机器人双层协作模型.该模型使协调决策和机器人运动成为2个功能独立的层次,使群体意图到个体行为的过度变为一个直接的过程,并在协调层通过采用Q学习算法在线学习不同状态下的最优策略,增强了决策系统的适应性和学习能力.在Q学习中通过把状态繁多的系统状态映射为为数不多的模糊状态,大大减少了状态空间的大小,避免了传统Q学习在状态空间和动作空间较大的情况下收敛速度慢,甚至不能收敛的缺点,提高了Q学习算法的收敛速度.最后,通过在足球机器人SimuroSot仿真比赛平台上进行实验,验证了双层协作模型的有效性.

With the conventional triple-layer decision-making model of soccer robots, decisions are sometimes inconsistent, leading to weaknesses in adaptability and self-learning ability. A double-layer cooperation model for a robot soccer system based on fuzzy Q-Learning is presented to solve these issues. This model divides cooperative decisions and robot movement into two layers with their own independent functions, so that the transition from group strategy to individual behavior becomes a direct process. To enhance the adaptability and self-learning capabilities of the decision-making system, the Q-learning algorithm was used in the cooperation layer to learn the optimal strategy for various conditions. To speed up the convergence of Q-learning and decrease the size of the state space, the numerous system states were mapped to seven fuzzy states in Q-learning. This avoids problems with Q-learning＇s slow converging rate when the size of the state space is large. This model was verified on the SimuroSot Robot Soccer Game platform.