人机对抗中的博弈学习方法

Game-Theoretic Learning in Human-Computer Gaming

近年来,人机对抗智能技术作为人工智能领域的前沿方向取得了一系列突破性的进展,如AlphaGo和DeepStack分别在围棋和二人无限注德州扑克中击败了人类专业选手.这些突破离不开博弈论和机器学习的深度结合.本文通过梳理当前人机对抗智能技术领域的重要工作,深入分析博弈论和机器学习在其中发挥的作用,总结了面向人机对抗任务的博弈学习研究框架,指出博弈论为人机对抗任务提供博弈模型和定义求解目标,机器学习帮助形成稳定高效可扩展的求解算法.具体地,本文首先介绍了人机对抗中的博弈学习方法的内涵,详细阐述了面向人机对抗任务的博弈学习研究框架,包括博弈模型构建、解概念定义、博弈解计算三个基本步骤,之后利用该框架分析了当前人机对抗智能技术领域的典型进展,最后指出了人机对抗中的博弈学习未来发展可能面临的挑战.本文梳理总结的人机对抗中的博弈学习研究框架为人机对抗智能技术领域的发展提供了方法保障和技术途径,同时也为通用人工智能的发展提供了新思路.

Recent development in the field of human-computer gaming,one of the frontiers in artificial intelligence(AI),has witnessed a series of breakthroughs,such as AlphaGo and DeepStack beat professional human players in Go and heads-up no-limit Texas Hold’em,respectively.Such successes demonstrate synergistic interactions between game theory and machine learning.Game theory is a theoretical framework that deals with strategic interactions among multiple rational players.Combined with machine learning,it is well suited for modeling,analyzing,and solving decision-making problems in human-computer gaming tasks that often involve two or more decision-makers.Game theory based learning methods thus receive increasing attention in recent years.Besides the popular multi-agent reinforcement learning approaches,there are some other game theory based learning methods,i.e.,game-theoretic learning methods,that are designed to converge to equilibria and can be dated back to the famous fictitious play proposed in 1951.In this paper,we give a selective overview of such game-theoretic learning methods in human-computer gaming.By analyzing key progresses in the field of human-computer gaming and game theory(including game-theoretic learning),we obtain a research framework for game-theoretic learning in human-computer gaming.In this framework,the role of game theory and machine learning each plays is identified:game theory provides models of strategic interactions and defines associated learning objectives(i.e.,solution concepts)while machine learning helps give rise to stable,efficient,and scalable game solving algorithms.In detail,we first review important progresses in the field of human-computer gaming and game theory.Then,we introduce the definition of game-theoretic learning in human-computer gaming and compare it with traditional machine learning methods such as supervised learning and single-agent reinforcement learning.After that,we elaborate on its research framework.Intuitively,this research framework equivalently or approximately transforms the problem of achieving a good performance in a class of human-computer gaming tasks into the problem of solving a class of games.As we summarize,such transformation usually takes three basic steps:game model formulation,solution concept definition,and game solution computation.Employing this framework,we also analyze a recent game-theoretic learning algorithm that combines fictitious play and deep reinforcement learning called neural fictitious self-play,and also three milestones in the field of human-computer gaming,i.e.,AlphaGo Zero,Libratus,and AlphaStar.At the end,we point out possible problems and challenges in the future research of game-theoretic learning in human-computer gaming,such as the definition of learning objectives in general-sum games,the interpretability of game-theoretic learning algorithms based on deep neural networks,the design of diverse environment suitable for game-theoretic learning,and the efficient solving of complex large-scale games that may exhibit non-transitive game behaviors.We believe that the research framework of game-theoretic learning in human-computer gaming offers guidance for the future development of human-computer gaming,and it also provides new perspectives on the development of artificial general intelligence.