基于观测空间关系提取的多智能体强化学习

Multi-agent reinforcement learning based on observation relation extraction

针对多智能体系统(multi-agent systems,MAS)中环境具有不稳定性、智能体决策相互影响所导致的策略学习困难的问题,提出了一种名为观测空间关系提取(observation relation extraction,ORE)的方法,该方法使用一个完全图来建模MAS中智能体观测空间不同部分之间的关系,并使用注意力机制来计算智能体观测空间不同部分之间关系的重要程度。通过将该方法应用在基于值分解的多智能体强化学习算法上,提出了基于观测空间关系提取的多智能体强化学习算法。在星际争霸微观场景(StarCraft multi-agent challenge,SMAC)上的实验结果表明,与原始算法相比,带有ORE结构的值分解多智能体算法在收敛速度和最终性能方面都有更好的性能。

In order to overcome the challenges of policy learning in MAS,such as the unstable environment and the interaction of agent decisions,this paper proposed a method named ORE,which used a complete graph to model the relationship between different parts of each agent’s observation,and took advantage of the attention mechanism to calculate the importance of the relationship between different parts of each agent’s observation.By applying the above method to multi-agent reinforcement learning algorithms based on value decomposition,this paper proposed multi-agent reinforcement learning algorithms based on observation relation extraction.Experimental results on SMAC show the proposed algorithms with ORE leads to better perfor-mance than the original algorithms in terms of both convergence speed and final performance.