脉冲强化学习算法研究综述

Research on Spiking Reinforcement Learning Algorithms:A Survey

借助人工神经网络(Artificial Neural Network,ANN),深度强化学习在游戏、机器人等复杂控制任务中取得了巨大的成功.然而,在认知能力与计算效率等方面,深度强化学习与大脑中的奖励学习机制相比仍存在着巨大的差距.受大脑中基于脉冲的通信方式启发,脉冲神经网络(Spiking Neural Network,SNN)使用拟合生物神经元机制的脉冲神经元模型进行计算,具有处理复杂时序数据的能力、极低的能耗以及较强的鲁棒性,并展现出了持续学习的潜力.在神经形态工程以及类脑计算领域中,SNN受到了广泛的关注,被誉为是新一代的神经网络.通过将SNN与强化学习相结合,脉冲强化学习算法被认为是发展人工大脑的一个可行途径,并能够有效解释生物大脑中的发现.作为神经科学与人工智能的交叉学科,脉冲强化学习算法涵盖了一大批杰出的研究工作.根据对不同领域的侧重,这些研究工作主要可以分为两大类:一类是以更好地理解大脑中的奖励学习机制为目的,用于解释动物实验中的发现,并对大脑学习进行仿真,例如R-STDP学习规则;另一类则是以实际控制任务中的性能、功耗等具体指标为导向,用作人工智能的一种鲁棒且低能耗的解决方案,在机器人、自主控制等领域具有巨大的应用潜力.本文首先介绍了脉冲强化学习算法的基础(即脉冲神经网络以及强化学习),然后对当前这两大类脉冲强化学习算法的研究特点与研究进展等进行分析.对于第一类算法,本文重点分析了利用三因素学习规则实现的强化学习算法,并回顾了其生理学背景以及具体实现方式.根据在训练过程中是否使用ANN,本文将第二类算法分为依托ANN实现的脉冲强化学习算法与基于脉冲的直接强化学习算法,并率先对这一脉冲强化学习算法的最新进展进行了系统性的梳理与分析,同时全面展示了在深度强化学习算法中应用SNN的不同方式.最后,本文对该领域的研究挑战以及后续研究方向进行了深入地探讨,总结了当前研究的优势与不足,并对其未来对神经科学以及人工智能领域可能产生的影响进行展望,以吸引更多研究人士参与这个新兴方向的交流与合作.

With the help of Artificial Neural Networks(ANNs),deep reinforcement learning algorithms have achieved great success in complex tasks,such as playing games and robotic control,etc.However,compared with the mechanism of reward-modulated learning in the brain,deep reinforcement learning still has a huge gap in cognitive ability and computational efficiency.Inspired by the spike-driven communication in the brain,Spiking Neural Networks(SNNs)adopt the spiking neuronal models for calculation and exchange information through discrete action potentials,i.e.,spikes,which greatly fit the mechanism of biological neurons.It is demonstrated by much research that SNNs have distinctive properties,such as complex time-series information processing capability,extremely low energy consumption,and strong robustness.In addition,SNNs also show the potential for continual learning.In the field of neuromorphic engineering and brain-inspired computing,SNNs are widely concerned and known as a new generation of neural networks.By combining SNNs with reinforcement learning,spiking reinforcement learning algorithms are considered as a feasible way to develop the artificial brain,and can effectively explain the discovery in the biological brain.As a cross-discipline research area of neuroscience and artificial intelligence,spiking reinforcement learning algorithms cover a large number of outstanding research works.According to the emphasis on different fields,these research works can be divided into two categories:one is to better understand the mechanism of rewardmodulated learning in the brain,which is used to explain the findings in animal experiments and simulate the learning process of the brain,such as R-STDP learning rules;The other is to improve the performance,energy efficiency and other specific indicators of various control tasks requiring reinforcement learning algorithms to solve.With the unique advantages of SNNs,this kind of algorithm acts as a robust and energy-efficient solution for artificial intelligence,and shows great application potential in the fields of robotics and autonomous control.In this paper,the first part presents the cornerstone of spiking reinforcement learning algorithms,that is,spiking neural networks and reinforcement learning,and then analyzes the research characteristics and progress of these two kinds of spiking reinforcement learning algorithms.For the first kind of algorithms,this paper focuses on analyzing the reinforcement learning algorithms using the three-factor learning rules,and introduces their physiological background and specific implementation methods.Based on whether to use ANNs during training,this paper further divides the second kind of algorithms into spiking reinforcement learning algorithms using ANNs and spike-based direct reinforcement learning algorithms.As far as we know,this paper takes the lead in systematically sorting out and analyzing the latest progress of spiking reinforcement learning algorithms,and comprehensively shows the different ways of applying SNNs in deep reinforcement learning algorithms.Finally,this paper makes an in-depth discussion of the current challenges and follow-up research directions in this field.We systematically summarize the advantages and disadvantages of the current research,and look forward to its future impact on the field of neuroscience and artificial intelligence,so as to attract more researchers to participate in communication and cooperation in this new direction.