稀疏奖励场景下基于个体落差情绪的多智能体协作算法

Multi-agent Cooperation Algorithm Based on Individual Gap Emotion in Sparse Reward Scenarios

针对在多智能体环境中强化学习面临的稀疏奖励问题,借鉴情绪在人类学习和决策中的作用,文中提出基于个体落差情绪的多智能体协作算法.对近似联合动作值函数进行端到端优化以训练个体策略,将每个智能体的个体动作值函数作为对事件的评估.预测评价与实际情况的差距产生落差情绪,以该落差情绪模型作为内在动机机制,为每个智能体产生一个内在情绪奖励,作为外在奖励的有效补充,以此缓解外在奖励稀疏的问题.同时内在情绪奖励与具体任务无关,因此具有一定的通用性.在不同稀疏程度的多智能体追捕场景中验证文中算法的有效性和鲁棒性.

To address the sparse reward problem confronted by reinforcement learning in multi-agent environment,a multi-agent cooperation algorithm based on individual gap emotion is proposed grounded on the role of emotions in human learning and decision making.The approximate joint action value function is optimized end-to-end to train individual policy,and the individual action value function of each agent is taken as an evaluation of the event.A gap emotion is generated via the gap between the predicted evaluation and the actual situation.The gap emotion model is regarded as an intrinsic motivation mechanism to generate an intrinsic emotion reward for each agent as an effective supplement to the extrinsic reward.Thus,the problem of sparse extrinsic rewards is alleviated.Moreover,the intrinsic emotional reward is task-independent and consequently it possesses some generality.The effectiveness and robustness of the proposed algorithm are verified in a multi-agent pursuit scenario with different sparsity levels.