A survey on multi-agent reinforcement learning and its application

Multi-agent reinforcement learning(MARL)has been a rapidly evolving field.This paper presents a comprehensive survey of MARL and its applications.We trace the historical evolution of MARL,highlight its progress,and discuss related survey works.Then,we review the existing works addressing inherent challenges and those focusing on diverse applications.Some representative stochastic games,MARL means,spatial forms of MARL,and task classification are revisited.We then conduct an in-depth exploration of a variety of challenges encountered in MARL applications.We also address critical operational aspects,such as hyperparameter tuning and computational complexity,which are pivotal in practical implementations of MARL.Afterward,we make a thorough overview of the applications of MARL to intelligent machines and devices,chemical engineering,biotechnology,healthcare,and societal issues,which highlights the extensive potential and relevance of MARL within both current and future technological contexts.Our survey also encompasses a detailed examination of benchmark environments used in MARL research,which are instrumental in evaluating MARL algorithms and demonstrate the adaptability of MARL to diverse application scenarios.In the end,we give our prospect for MARL and discuss their related techniques and potential future applications.

Multi-agent system application in accordance with game theory in bi-directional coordination network model

The multi-agent system is the optimal solution to complex intelligent problems. In accordance with the game theory, the concept of loyalty is introduced to analyze the relationship between agents' individual income and global benefits and build the logical architecture of the multi-agent system. Besides, to verify the feasibility of the method, the cyclic neural network is optimized, the bi-directional coordination network is built as the training network for deep learning, and specific training scenes are simulated as the training background. After a certain number of training iterations, the model can learn simple strategies autonomously. Also,as the training time increases, the complexity of learning strategies rises gradually. Strategies such as obstacle avoidance, firepower distribution and collaborative cover are adopted to demonstrate the achievability of the model. The model is verified to be realizable by the examples of obstacle avoidance, fire distribution and cooperative cover. Under the same resource background, the model exhibits better convergence than other deep learning training networks, and it is not easy to fall into the local endless loop.Furthermore, the ability of the learning strategy is stronger than that of the training model based on rules, which is of great practical values.

A Game Theoretic Approach for a Minimal Secure Dominating Set

The secure dominating set(SDS),a variant of the dominating set,is an important combinatorial structure used in wireless networks.In this paper,we apply algorithmic game theory to study the minimum secure dominating set(Min SDS) problem in a multi-agent system.We design a game framework for SDS and show that every Nash equilibrium(NE) is a minimal SDS,which is also a Pareto-optimal solution.We prove that the proposed game is an exact potential game,and thus NE exists,and design a polynomial-time distributed local algorithm which converges to an NE in O(n) rounds of interactions.Extensive experiments are done to test the performance of our algorithm,and some interesting phenomena are witnessed.

A single-task and multi-decision evolutionary game model based on multi-agent reinforcement learning

In the evolutionary game of the same task for groups,the changes in game rules,personal interests,the crowd size,and external supervision cause uncertain effects on individual decision-making and game results.In the Markov decision framework,a single-task multi-decision evolutionary game model based on multi-agent reinforcement learning is proposed to explore the evolutionary rules in the process of a game.The model can improve the result of a evolutionary game and facilitate the completion of the task.First,based on the multi-agent theory,to solve the existing problems in the original model,a negative feedback tax penalty mechanism is proposed to guide the strategy selection of individuals in the group.In addition,in order to evaluate the evolutionary game results of the group in the model,a calculation method of the group intelligence level is defined.Secondly,the Q-learning algorithm is used to improve the guiding effect of the negative feedback tax penalty mechanism.In the model,the selection strategy of the Q-learning algorithm is improved and a bounded rationality evolutionary game strategy is proposed based on the rule of evolutionary games and the consideration of the bounded rationality of individuals.Finally,simulation results show that the proposed model can effectively guide individuals to choose cooperation strategies which are beneficial to task completion and stability under different negative feedback factor values and different group sizes,so as to improve the group intelligence level.

Optimal Time-of-use Pricing for Renewable Energy-powered Microgrids: A Multi-agent Evolutionary Game Theory-based Approach

While price schedules can help improve the economic efficiency of renewable energy-powered microgrids,timeof-use(TOU)pricing has been identified as an effective way for microgrid development,which is presently limited by its high costs.In this study,we propose an evolutionary game theoretic model to explore optimal TOU pricing for development of renewable energy-powered microgrids by applying a multi-agent system,that comprises a government agent,local utility company agent,and different types of consumer agents.In the proposed model,we design objective functions for the company and the consumers and obtain a Nash equilibrium using backward induction.Two pricing strategies,namely,the TOU seasonal pricing and TOU monthly pricing,are evaluated and compared with traditional fixed pricing.The numerical results demonstrate that TOU schedules have significant potential for development of renewable energy-powered microgrids and are recommended for an electric company to replace traditional fixed pricing.Additionally,TOU monthly pricing is more suitable than TOU seasonal pricing for microgrid development.

A Multi-Agent Approach for Solving Traveling Salesman Problem

The traveling salesman problem （TSP） is a classical optimization problem and it is one of a class of NP- Problem. This paper presents a new method named multiagent approach based genetic algorithm and ant colony system to solve the TSP. Three kinds of agents with different function were designed in the multi-agent architecture proposed by this paper. The first kind of agent is ant colony optimization agent and its function is generating the new solution continuously. The second kind of agent is selection agent, crossover agent and mutation agent, their function is optimizing the current solutions group. The third kind of agent is fast local searching agent and its function is optimizing the best solution from the beginning of the trial. At the end of this paper, the experimental results have shown that the proposed hybrid ap proach has good performance with respect to the quality of solution and the speed of computation.

Consensus of high-order dynamic multi-agent systems with switching topology and time-varying delays

This paper studies the consensus problems for a group of agents with switching topology and time-varying communication delays, where the dynamics of agents is modeled as a high-order integrator. A linear distributed consensus protocol is proposed, which only depends on the agent＇s own information and its neighbors＇ partial information. By introducing a decomposition of the state vector and performing a state space transformation, the closed-loop dynamics of the multi-agent system is converted into two decoupled subsystems. Based on the decoupled subsystems, some sufficient conditions for the convergence to consensus are established, which provide the upper bounds on the admissible communication delays. Also, the explicit expression of the consensus state is derived. Moreover, the results on the consensus seeking of the group of high-order agents have been extended to a network of agents with dynamics modeled as a completely controllable linear time-invariant system. It is proved that the convergence to consensus of this network is equivalent to that of the group of high-order agents. Finally, some numerical examples are given to demonstrate the effectiveness of the main results.

An Application of Game Teaching Approach in Primary School English Vocabulary Teaching——A Case Study of Wenzhou Jingshan Primary School

Vocabulary is a key to English learning. How to memorize words effectively has become a heated topic among teachers and students. Since primary school is the initial stage of English learning, it is imperative to cultivate students' interest in English and find a useful way of learning vocabulary. The study aims to prove that game teaching approach as a commonly used teaching approach can motivate students' interest and assist English teaching if it is combined with vocabulary teaching properly. In the study, 70 students of Grade four were randomly chosen from Wenzhou Jingshan primary school as the subjects, class 1as the experimental group while Class 2 as the control one. The results show that game teaching approach applied in English vocabulary teaching can help students learn vocabulary more efficiently than traditional vocabulary teaching method.

An Empirical Study on Google Research Football Multi-agent Scenarios

Few multi-agent reinforcement learning (MARL) researches on Google research football (GRF) focus on the 11-vs-11 multi-agent full-game scenario and to the best of our knowledge, no open benchmark on this scenario has been released to the public. In this work, we fill the gap by providing a population-based MARL training pipeline and hyperparameter settings on multi-agent football scenario that outperforms the bot with difficulty 1.0 from scratch within 2 million steps. Our experiments serve as a reference for the expected performance of independent proximal policy optimization (IPPO), a state-of-the-art multi-agent reinforcement learning algorithm where each agent tries to maximize its own policy independently across various training configurations. Meanwhile, we release our training framework Light-MALib which extends the MALib codebase by distributed and asynchronous implementation with additional analytical tools for football games. Finally, we provide guidance for building strong football AI with population-based training and release diverse pretrained policies for benchmarking. The goal is to provide the community with a head start for whoever experiment their works on GRF and a simple-to-use population-based training framework for further improving their agents through self-play. The implementation is available at https://github.com/Shanghai-Digital-Brain-Laboratory/DB-Football.

中医药文化传承研究——网游世界新视角

元宇宙时代,网络游戏不仅是人们的重要娱乐方式,更是网游爱好者的精神乐园。对青少年的行为方式、思维习惯、精神面貌产生了巨大影响。同时,网络游戏也已逐渐变身为一个新型的“文化载体”,依靠该平台的巨大吸引力,寓传统文化传承创新于游戏娱乐之中,将有助于推动游戏在青少年成长中发挥积极、正向的引领作用。国家对中医药事业发展及数字媒体愈发重视,为更好地响应国家号召,宣传和弘扬中医药文化知识,文章从“使用与满足”理论层面阐释了网络游戏结合中华传统文化传承的现实路径,并以中医药文化为蓝本,创新现代网络游戏设计,以挖掘出新时代背景下中医药文化传承的嬗变之路。

高中英语“Quiz Game”教学法的实践研究

外语教育思想及模式应作与时俱进的改革,应由注重语言形式教学转向掌握语言形式和培养交流能力并重。本课题旨在探讨运用到课堂是实现任务学习法教学的目的的唯一正确途径,切入此途径是教师角色的转换。“Quiz Game”的课堂组织形式是进入竞赛式活动型互动交际课堂的具体操作方式。

基于博弈论与模糊区间理论的森林康养基地评价方法

为了提高森林康养基地等级划分的精度,对相同等级下不同地区之间的优劣做出比较,提出了一种基于博弈论与模糊区间理论的森林康养基地评价方法,建立了区间型森林康养基地评价指标体系.利用博弈论将模糊熵与变异系数权重进行组合,借助模糊区间理论构建评价模型,并采用格贴近度公式得到待评价地区隶属的等级,引入TOPSIS模型对所选取的待评价地区进行优劣排序.结果表明,该评价方法具有良好的适用性与有效性.

Collision-free parking recommendation based on multi-agent reinforcement learning in vehicular crowdsensing

The recent proliferation of Fifth-Generation(5G)networks and Sixth-Generation(6G)networks has given rise to Vehicular Crowd Sensing(VCS)systems which solve parking collisions by effectively incentivizing vehicle participation.However,instead of being an isolated module,the incentive mechanism usually interacts with other modules.Based on this,we capture this synergy and propose a Collision-free Parking Recommendation(CPR),a novel VCS system framework that integrates an incentive mechanism,a non-cooperative VCS game,and a multi-agent reinforcement learning algorithm,to derive an optimal parking strategy in real time.Specifically,we utilize an LSTM method to predict parking areas roughly for recommendations accurately.Its incentive mechanism is designed to motivate vehicle participation by considering dynamically priced parking tasks and social network effects.In order to cope with stochastic parking collisions,its non-cooperative VCS game further analyzes the uncertain interactions between vehicles in parking decision-making.Then its multi-agent reinforcement learning algorithm models the VCS campaign as a multi-agent Markov decision process that not only derives the optimal collision-free parking strategy for each vehicle independently,but also proves that the optimal parking strategy for each vehicle is Pareto-optimal.Finally,numerical results demonstrate that CPR can accomplish parking tasks at a 99.7%accuracy compared with other baselines,efficiently recommending parking spaces.

On Aspiration Solutions in Predicting Coalition Formationin Cooperative Games

This paper consists of two parts. The first part introduces the strict aspiration as a new aspiration solution concept, which is provedto be existent for any cooperative game. The second part deals with theunsolved problem put forward by Bennett by showing that there is atleast one payoff which is balanced, partnered and equal gains aspiration.The proof is algebraic and constructive, thus providing an algorithm forfinding such aspirations.

Modeling and Analysis of the Decentralized Interactive Cyber Defense Approach

Powered by the Internet and the ever-increasing level of informatization, the cyberspace has become increasingly complex and its security situation has become increasingly grim, which requires new adaptive and collaborative defense technologies. In this paper, we introduced an extended interactive multi-agent decision model for decentralized cyber defense. Based on the significant advantages of the cooperative multi-agent decision-making, the decentralized interactive decision model DI-MDPs and the corresponding interaction and retrieval algorithms are proposed. Then, we analyzed the interactive decision by the calculation and update processes of three matrices, the stability and evolutionary equilibrium of the proposed model are also analyzed. Finally, we evaluated the performance of the proposed algorithms based on open data sets and standard test environments, the experimental results shown that the proposed work will be more applicable in cyber defense.

基于演化博弈论的可变导向车道设置策略研究

可变导向车道的设置,在提高道路通行效率的同时会造成驾驶人情绪紧张或者因不了解路况导致违规甚至发生事故,对交通系统的安全和稳定运行产生负面影响。以交管部门设置可变导向车道的策略选择为研究目标,结合驾驶人走可变导向车道时产生的紧张情绪损失等因素,构建交管部门-驾驶人群体两方博弈模型。采用演化博弈论的方法分析博弈双方选择行为演化和稳定策略,对系统中存在的5个演化均衡点进行分析,得到系统的3种理想稳定状态和相对应的参数条件,为交管部门设置可变导向车道进行策略选择提供理论支撑。

基于多智能体博弈的路网旅客列车运行图协调优化研究

为实现跨线列车与本线列车运行线协调编制,提升列车运行图编制质量及自动化编制水平,提出基于多智能体博弈的路网旅客列车运行图协调优化方法。首先,以跨线列车和本线列车开行效益加权总和最大化为目标,以车站能力和列车运行间隔时间等为约束,构建基于多智能体博弈的路网旅客列车运行图协调优化模型。其次,以“控制纵向调整范围,改变横向调整幅度”为原理设计算法求解模型。最后,以京沪高铁6:00—12:00时段的列车运行图为例验证模型及算法的有效性,结果表明,运用文中所提方法可以在不同情况下得出跨线列车与本线列车运行线的冲突消解策略,实现路网旅客列车运行图协调优化。

经济数字化背景下国际税收规则变革:方案评析、内在逻辑与中国路径

经济数字化背景下国际税收改革成为世界百年未有之大变局之际的重点议题之一。近年来税基侵蚀和利润转移、跨境税源分割失衡加剧等新挑战凸显,倒逼国际税收规则变革和国际税收秩序重塑。市场国“单边征税先行”、联合国试图“双边协商”和OECD推动下的“多边合作共识”共同刻画了当前数字经济时代国际税收规则变革的现实图景。文章基于马克思主义价值创造理论和国家竞争理论,深入剖析经济数字化国际税收规则变革的内在逻辑,解析数字经济居民国与市场国、发达国家与发展中国家展开的多重博弈。立足于中国基本国情,文章提出战略层面积极有为参与全球税收治理建设,制度层面科学有效制定中国应对策略,技术层面持续提升数字化税收治理能力,以期推动形成更具包容性的国际税收秩序。

P2P transaction method for distributed energy prosumers based on reputation value

Adding a reputation incentive system to peer-to-peer(P2P)energy transactions can encourage prosumers to regulate their trading behavior,which is important for ensuring the efficiency and reliability of P2P transactions.This study proposed a P2P transaction mechanism and game optimization model for prosumers involved in distributed energy sources considering reputation-value incentives.First,the deviation of P2P transactions and the non-consumption rate of distributed renewable energy in P2P transactions were established as indicators to quantify the influencing factors of the reputation value,and a reputation incentive model of P2P transactions for prosumers was constructed.Then,the penalty coefficient was applied to the cost function of the prosumers,and a non-cooperative game model of P2P transactions based on the complete information of multi-prosumers was established.Furthermore,the Nash equilibrium problem was transformed into a nonlinear optimization problem by constructing the modified optimal reaction function,and the Nash equilibrium solution of the game was obtained via a relaxation algorithm.Finally,the modified IEEE 33-node test system based on electricity market P2P and an IEEE 123-node test system were used to analyze and verify the cost and P2P participation of prosumers considering the reputation value.The results show that the addition of the reputation incentive system can encourage prosumers to standardize their interactive transaction behavior and actively participate in P2P transactions.It can also improve the operation efficiency of the power grid and promote the perfection of the P2P transaction mechanism.