A survey on multi-agent reinforcement learning and its application

Multi-agent reinforcement learning(MARL)has been a rapidly evolving field.This paper presents a comprehensive survey of MARL and its applications.We trace the historical evolution of MARL,highlight its progress,and discuss related survey works.Then,we review the existing works addressing inherent challenges and those focusing on diverse applications.Some representative stochastic games,MARL means,spatial forms of MARL,and task classification are revisited.We then conduct an in-depth exploration of a variety of challenges encountered in MARL applications.We also address critical operational aspects,such as hyperparameter tuning and computational complexity,which are pivotal in practical implementations of MARL.Afterward,we make a thorough overview of the applications of MARL to intelligent machines and devices,chemical engineering,biotechnology,healthcare,and societal issues,which highlights the extensive potential and relevance of MARL within both current and future technological contexts.Our survey also encompasses a detailed examination of benchmark environments used in MARL research,which are instrumental in evaluating MARL algorithms and demonstrate the adaptability of MARL to diverse application scenarios.In the end,we give our prospect for MARL and discuss their related techniques and potential future applications.

Analysis of Commuting Modal Shift in Consideration of Social Interaction of Consciousness for Environment

It is the matter for achievement of the low carbon transport system that the excessive use of private vehicles can be controlled appropriately.Not only improvement of service level of modes except private vehicle,but also consciousness for environmental problem of individual trip maker is important for eco-commuting promotion.On the other hand,consciousness for environment would be changed by influence of other person.Accordingly,it is aimed in the study that the structure of decision-making process for modal shift to the eco-commuting mode in the local city is described considering environmental consciousness and social interaction.For the purpose,the consciousness for the environment problem and the travel behavior of the commuter at the suburban area in the local city are investigated by the questionnaire survey.The covariance structure about the eco-consciousness is analyzed with the database of the questionnaire survey by structural equation modeling.As the result,it can be confirmed with the structural equation model that the individual environmental consciousness is strongly related with the intention of self-sacrifice and is influenced with the local interaction of the individual connections.On the other hand,the intention of modal shift for the commuting mode is analyzed with the database of the questionnaire survey.It can be found out that the environmental consciousness is not statistically significant for commuting mode choice with the present poor level of service of public transport.However,the intention of self-sacrifice for the prevention of the global warming is statistically confirmed as the factor of modal shift with the operation of eco-commuting bus service with the RP/SP integrated estimation method.As the result,the multi-agent simulation system with social interaction model for eco consciousness is developed to measure the effect of the eco-commuting promotion.For the purpose,the carbon dioxide emission is estimated based on traffic demand and road network condition in the traffic environment model.On the other hand,the relation between agents is defined based on the small world network.The proposed multi-agent simulation is applied to measure the effect of the eco-commuting promotion such as improvement of level of service on the public transport or education of eco-consciousness.The effect of the promotion plan can be observed with the proposed multi-agent system.Finally,it can be concluded that the proposed multi-agent simulation with social interaction for eco-consciousness is useful for planning of eco-commuting promotion.

A Multi-Agent System for Environmental Monitoring Using Boolean Networks and Reinforcement Learning

Distributed wireless sensor networks have been shown to be effective for environmental monitoring tasks,in which multiple sensors are deployed in a wide range of the environments to collect information or monitor a particular event,Wireless sensor networks,consisting of a large number of interacting sensors,have been successful in a variety of applications where they are able to share information using different transmission protocols through the communication network.However,the irregular and dynamic environment requires traditional wireless sensor networks to have frequent communications to exchange the most recent information,which can easily generate high communication cost through the collaborative data collection and data transmission.High frequency communication also has high probability of failure because of long distance data transmission.In this paper,we developed a novel approach to multi-sensor environment monitoring network using the idea of distributed system.Its communication network can overcome the difficulties of high communication cost and Single Point of Failure(SPOF)through the decentralized approach,which performs in-network computation.Our approach makes use of Boolean networks that allows for a non-complex method of corroboration and retains meaningful information regarding the dynamics of the communication network.Our approach also reduces the complexity of data aggregation process and employee a reinforcement learning algorithm to predict future event inside the environment through the pattern recognition.

Multi-agent-based Approach for Determination of Time-quota in Integrated Environment

Time-quota is one of important factors in producti on system. It is affected by various factors. time-quota is studied in CAPP and p roduction schedule integration environment in this paper. An agent-based time- quota method is put forward and the structure model is established by means of i ntelligent agent in integrated environment. The method can map the influencing t ime-quota factors into part agent related to process state and machine method a gent, resorting to the function of agent rule-based reasoning, the agents can t ransform these factors into data mode that artificial neural network (ANN) can a ccept and recognize. As a tool, ANN agent can calculate time-quota quickly. A b lackboard method is used as the means of communication and collaborative control between agents. The experiments show that precise process time-quota can be obtained rapidly with proper samples selected, continuous self-study and self -organization in system, and multi-agent approach is an effective method for d etermination of time-quota.

Multi-Agent Dynamic Area Coverage Based on Reinforcement Learning with Connected Agents

Dynamic area coverage with small unmanned aerial vehicle(UAV)systems is one of the major research topics due to limited payloads and the difficulty of decentralized decision-making process.Collaborative behavior of a group of UAVs in an unknown environment is another hard problem to be solved.In this paper,we propose a method for decentralized execution of multi-UAVs for dynamic area coverage problems.The proposed decentralized decision-making dynamic area coverage(DDMDAC)method utilizes reinforcement learning(RL)where each UAV is represented by an intelligent agent that learns policies to create collaborative behaviors in partially observable environment.Intelligent agents increase their global observations by gathering information about the environment by connecting with other agents.The connectivity provides a consensus for the decision-making process,while each agent takes decisions.At each step,agents acquire all reachable agents’states,determine the optimum location for maximal area coverage and receive reward using the covered rate on the target area,respectively.The method was tested in a multi-agent actor-critic simulation platform.In the study,it has been considered that each UAV has a certain communication distance as in real applications.The results show that UAVs with limited communication distance can act jointly in the target area and can successfully cover the area without guidance from the central command unit.

Multi-agent System Optimized Reconfiguration of Shipboard Power System

Reconfigurability of the electrical network in a shipboard power system （SPS） after its failure is central to the restoration of power supply and improves survivability of an SPS. The navigational process creates a sequence of different operating conditions. The priority of some loads differs in changing operating conditions. After analyzing characteristics of typical SPS, a model was developed used a grade III switchboard and an environmental prioritizing agent （EPA） algorithm. This algorithm was chosen as it is logically and physically decentralized as well as multi-agent oriented. The EPA algorithm was used to decide on the dynamic load priority, then it selected the means to best meet the maximum power supply load. The simulation results showed that higher priority loads were the first to be restored. The system satisfied all necessary constraints, demonstrating the effectiveness and validity of the proposed method.

Classifying Environmental Features From Local Observations of Emergent Swarm Behavior

Robots in a swarm are programmed with individual behaviors but then interactions with the environment and other robots produce more complex, emergent swarm behaviors. One discriminating feature of the emergent behavior is the local distribution of robots in any given region. In this work, we show how local observations of the robot distribution can be correlated to the environment being explored and hence the location of openings or obstructions can be inferred. The correlation is achieved here with a simple, single-layer neural network that generates physically intuitive weights and provides a degree of robustness by allowing for variation in the environment and number of robots in the swarm. The robots are simulated assuming random motion with no communication, a minimalist model in robot sophistication, to explore the viability of cooperative sensing. We culminate our work with a demonstration of how the local distribution of robots in an unknown, office-like environment can be used to locate unobstructed exits.

Using GIS Data to Build Informed Virtual Geographic Environments (IVGE)

In this paper, we propose a novel approach to automatically building Informed Virtual Geographic Environments (IVGE) using data provided by Geographic Information Systems (GIS). The obtained IVGE provides 2D and 3D geographic information for visualization and simulation purposes. Conventional VGE approaches are generally built upon a grid-based representation, raising the well-known problems of the lack of accuracy of the localized data and the difficulty to merge data with multiple semantics. On the contrary, our approach uses a topological model and provides an exact representation of GIS data, allowing an accurate geometrical exploitation. Moreover, our model can merge semantic information, even if spatially overlapping. In addition, the proposed IVGE contains spatial information which can be enhanced thanks to a geometric abstraction method. We illustrate this model with an application which automatically extracts the required data from standard GIS files and allows a user to navigate and retrieve information from the computed IVGE.

A Multi-Agent Expert System for Prevention of Child Abuse and Neglect

When a child abuse situation arises, the responsible agencies and entities in charge of response should be capable of providing a fast and personalized solution for the good of the child. This need leads the authors to consider the formation of dynamic virtual organizations tailored to each particular abuse case. In the authors＇ approach the partner selection of these collaborative networks is done through a software tool that combines two technologies from the field of artificial intelligence, specifically multi-agent systems and expert systems. In addition, these partners come from the breeding environment constituted by all the agencies or individuals, either in a region or locality, which have the potential of response.

A Multi-agent Organization Approach for Developing Social-Technical Software of Autonomous Robots

Software of autonomous robot is a complex physical and social technical system that is context-aware, autonomous and capable of self-management to achieve tasks. It typically consists of a large amount of autonomous entities and interactions. To develop such system needs high-level metaphors and effective mechanisms independent of physical and technical details of various robots.The paper presents a multi-agent organization approach to developing autonomous robot software that is modelled as social organization, in which each agent is bound to specific roles with specified responsibilities that are tightly related with robot’s characteristics and tasks. These agents form diverse organization structure and patterns to achieve flexible cooperation in order to achieve assigned tasks. The paper details multi-agent organization model of autonomous robot software and various roles in the model. We have implemented a framework called AutoRobot that realizes the approach and supports the development and running of autonomous robot software. A case is studied by using NAO robot to show the effectiveness of our proposed approach.

多主体系统服务模型的研究

在研究多主体系统理论的基础上,结合面向服务的体系结构,提出了一个多主体系统的服务模型,给出了该服务模型的体系结构,能有效整合异构服务资源,解决用户服务请求,在多主体环境MAGE上进行实现,并在城市应急联动与社会综合服务系统中得到了应用。

Adaptive Execution of Jobs in Computational Grid Environment

In a computational grid, jobs must adapt to the dynamically changing heterogeneous environment with an objective of maintaining the quality of service. In order to enable adaptive execution of multiple jobs running concurrently in a computational grid, we propose an integrated performance-based resource management framework that is supported by a multi-agent system （MAS）. The multi-agent system initially allocates the jobs onto different resource providers based on a resource selection algorithm. Later, during runtime, if performance of any job degrades or quality of service cannot be maintained for some reason （resource failure or overloading）, the multi-agent system assists the job to adapt to the system. This paper focuses on a part of our framework in which adaptive execution facility is supported. Adaptive execution facility is availed by reallocation and local tuning of jobs. Mobile, as well as static agents are employed for this purpose. The paper provides a summary of the design and implementation and demonstrates the efficiency of the framework by conducting experiments on a local grid test bed.

Using emotions for the development of human-agent societies

Human-agent societies refer to applications where virtual agents and humans coexist and interact transparently into a fully integrated environment. One of the most important aspects in this kind of applications is including emotional states of the agents(humans or not) in the decision-making process. In this sense, this paper presents the applicability of the JaCalIVE(Jason Cartago implemented intelligent virtual environment) framework for developing this kind of society. Specifically, the paper presents an ambient intelligence application where humans are immersed into a system that extracts and analyzes the emotional state of a human group. A social emotional model is employed to try to maximize the welfare of those humans by playing the most appropriate music in every moment.

Mobile sensors’patrol path planning in unobservable border region

Purpose-The border control becomes challenging when a protected region is large and there is a limited number of border patrols.This research paper proposes a novel heuristic-based patrol path planning scheme in order to efficiently patrol with resource scarcity.Design/methodology/approach-The trespasser influencing score,which is determined from the environmental characteristics and trespassing statistic of the region,is used as a heuristic for measuring a chance of approaching a trespasser.The patrol plan is occasionally updated with a new trespassing statistic during a border operation.The performance of the proposed patrol path planning scheme was evaluated and compared with other patrol path planning schemes by the empirical experiment under different scenarios.Findings-The result from the experiment indicates that the proposed patrol planning outperforms other patrol path planning schemes in terms of the trespasser detection rate,when more environment-aware trespassers are in the region.Research limitations/implications-The experiment was conducted through simulated agents in simulated environment,which were assumed to mimic real behavior and environment.Originality/value-This research paper contributes a heuristic-based patrol path planning scheme that applies the environmental characteristics and dynamic statistic of the region,as well as a border surveillance problem model that would be useful for mobile sensor planning in a border surveillance application.