深度逆向强化学习研究综述

Overview of deep inverse reinforcement learning

深度逆向强化学习是机器学习领域的一个新的研究热点,它针对深度强化学习的回报函数难以获取问题,提出了通过专家示例轨迹重构回报函数的方法。首先介绍了3类深度强化学习方法的经典算法;接着阐述了经典的逆向强化学习算法,包括基于学徒学习、最大边际规划、结构化分类和概率模型形式化的方法;然后对深度逆向强化学习的一些前沿方向进行了综述,包括基于最大边际法的深度逆向强化学习、基于深度Q网络的深度逆向强化学习和基于最大熵模型的深度逆向强化学习和示例轨迹非专家情况下的逆向强化学习方法等。最后总结了深度逆向强化学习在算法、理论和应用方面存在的问题和发展方向。

Deep inverse reinforcement learning is a new research hotspot in the field of machine learning. It aims at recovering the reward function of deep reinforcement learning by the experts' example trajectories. This paper systematically introduces three kinds of classic deep reinforcement learning methods. Then inverse reinforcement learning algorithms including apprenticeship learning, max margin plan, structured classification and probability models are described; then,some frontier researches of deep inverse reinforcement learning are reviewed, including the deep max margin plan inverse reinforcement learning, deep inverse reinforcement learning based on DQN and deep maximum entropy inverse reinforcement learning and recovering reward functions from non-expert trajectories etc. Finally, the existing issues and development direction are summarized.