逆向强化学习研究综述

Survey on Inverse Reinforcement Learning

逆向强化学习(inverse reinforcement learning,IRL)也称为逆向最优控制(inverse optimal control,IOC),是强化学习和模仿学习领域的一种重要研究方法,该方法通过专家样本求解奖赏函数,并根据所得奖赏函数求解最优策略,以达到模仿专家策略的目的.近年来,逆向强化学习在模仿学习领域取得了丰富的研究成果,已广泛应用于汽车导航、路径推荐和机器人最优控制等问题中.首先介绍逆向强化学习理论基础,然后从奖赏函数构建方式出发,讨论分析基于线性奖赏函数和非线性奖赏函数的逆向强化学习算法,包括最大边际逆向强化学习算法、最大熵逆向强化学习算法、最大熵深度逆向强化学习算法和生成对抗模仿学习等.随后从逆向强化学习领域的前沿研究方向进行综述,比较和分析该领域代表性算法,包括状态动作信息不完全逆向强化学习、多智能体逆向强化学习、示范样本非最优逆向强化学习和指导逆向强化学习等.最后总结分析当前存在的关键问题,并从理论和应用方面探讨未来的发展方向.

Inverse reinforcement learning(IRL),also known as inverse optimal control(IOC),is an important research method of reinforcement learning and imitation learning.IRL solves a reward function from expert samples,and the optimal strategy is then solved to imitate expert strategies.In recent years,fruitful achievements have been yielded by IRL in imitation learning,with widespread application in vehicle navigation,path recommendation,and robotic optimal control.First,this study presents the theoretical basis of IRL.Then,from the perspective of reward function construction methods,IRL algorithms based on linear and non-linear reward functions are analyzed.The algorithms include maximum marginal IRL,maximum entropy IRL,maximum entropy deep IRL,and generative adversarial imitation learning.In addition,frontier research directions of IRL are reviewed to compare and analyze relevant representative algorithms containing IRL with incomplete expert demonstrations,multi-agent IRL,IRL with sub-optimal expert demonstrations,and guiding IRL.Finally,the primary challenges of IRL and future developments in its theoretical and application significance are summarized.