Cooperative multi-target hunting by unmanned surface vehicles based on multi-agent reinforcement learning

To solve the problem of multi-target hunting by an unmanned surface vehicle(USV)fleet,a hunting algorithm based on multi-agent reinforcement learning is proposed.Firstly,the hunting environment and kinematic model without boundary constraints are built,and the criteria for successful target capture are given.Then,the cooperative hunting problem of a USV fleet is modeled as a decentralized partially observable Markov decision process(Dec-POMDP),and a distributed partially observable multitarget hunting Proximal Policy Optimization(DPOMH-PPO)algorithm applicable to USVs is proposed.In addition,an observation model,a reward function and the action space applicable to multi-target hunting tasks are designed.To deal with the dynamic change of observational feature dimension input by partially observable systems,a feature embedding block is proposed.By combining the two feature compression methods of column-wise max pooling(CMP)and column-wise average-pooling(CAP),observational feature encoding is established.Finally,the centralized training and decentralized execution framework is adopted to complete the training of hunting strategy.Each USV in the fleet shares the same policy and perform actions independently.Simulation experiments have verified the effectiveness of the DPOMH-PPO algorithm in the test scenarios with different numbers of USVs.Moreover,the advantages of the proposed model are comprehensively analyzed from the aspects of algorithm performance,migration effect in task scenarios and self-organization capability after being damaged,the potential deployment and application of DPOMH-PPO in the real environment is verified.

Regional Multi-Agent Cooperative Reinforcement Learning for City-Level Traffic Grid Signal Control

This article studies the effective traffic signal control problem of multiple intersections in a city-level traffic system.A novel regional multi-agent cooperative reinforcement learning algorithm called RegionSTLight is proposed to improve the traffic efficiency.Firstly a regional multi-agent Q-learning framework is proposed,which can equivalently decompose the global Q value of the traffic system into the local values of several regions Based on the framework and the idea of human-machine cooperation,a dynamic zoning method is designed to divide the traffic network into several strong-coupled regions according to realtime traffic flow densities.In order to achieve better cooperation inside each region,a lightweight spatio-temporal fusion feature extraction network is designed.The experiments in synthetic real-world and city-level scenarios show that the proposed RegionS TLight converges more quickly,is more stable,and obtains better asymptotic performance compared to state-of-theart models.

An Improved Bounded Conflict-Based Search for Multi-AGV Pathfinding in Automated Container Terminals

As the number of automated guided vehicles(AGVs)within automated container terminals(ACT)continues to rise,conflicts have becomemore frequent.Addressing point and edge conflicts ofAGVs,amulti-AGVconflict-free path planning model has been formulated to minimize the total path length of AGVs between shore bridges and yards.For larger terminalmaps and complex environments,the grid method is employed to model AGVs’road networks.An improved bounded conflict-based search(IBCBS)algorithmtailored to ACT is proposed,leveraging the binary tree principle to resolve conflicts and employing focal search to expand the search range.Comparative experiments involving 60 AGVs indicate a reduction in computing time by 37.397%to 64.06%while maintaining the over cost within 1.019%.Numerical experiments validate the proposed algorithm’s efficacy in enhancing efficiency and ensuring solution quality.

Trading platform for cooperation and sharing based on blockchain within multi-agent energy internet

With the release of the electricity sales side,large-scale small-capacity distributed power generation units are connected to the distribution side,forming multi-type market entities such as microgrids,integrated energy systems,and virtual power plants.With the large-scale integration of distributed energy,the energy market under the energy internet is different from a traditional transmission grid.It is currently developing in the direction of diversified entities and commodities,a flat structure,and a flexible and competitive multi-agent market mechanism.In this context,this study analyzes the value of combining blockchain and the electricity market presents the design of a blockchain trading framework for multi-agent cooperation and sharing of the energy internet.The nodes in market transactions are modeled through power system modeling in the physical layer and the transaction consensus strategy in the cyber layer;moreover,the nodes are verified in a modified IEEE 13 testing feeder of a distribution network.A transaction example is demonstrated using the multi-agent cooperation and sharing transaction platform based on the Ethereum private blockchain.

Rough set based multi-agent system cooperation for industrial supervisory interface system

In this paper, rough set theory is introduced into the interface multi-agent system （MAS） for industrial supervisory system. Taking advantages of rough set in data mining, a cooperation model for MAS is built. Rules for avoiding cooperation conflict are deduced. An optimization algorithm is used to enhance security and real time attributes of the system. An application based on the proposed algorithm and rules are given.

Multi-agent based modeling for order distribution cooperative negotiation in supply chains

With the new characteristics of global cooperation in supply chains being synthetically considered,a hybrid model to the cooperative negotiation process for the order distribution in supply chain is mainly studied.After reviewing and analyzing some main domestic and overseas processes in cooperative negotiation modeling in supply chain,some problems are subsequently pointed out.For example,the traditional simple multi-agent system(MAS)frameworks which have some limitations,are not suitable for solving modeling complex systems.To solve these problems,thinking with the aid of the multi-agent structure and complex system modeling,the manufacturing supply chain is taken as an example,and a time Petri net production model is adopted to decompose the materials.And then a cooperative negotiation model for the order distribution in supply chain is constructed based on combining multi-agent techniques with time Petri net modeling.The simulation results reveal that the above model helps solve the problems of cooperative negotiation in supply chains.

Immune multi-agent model using vaccine for cooperative air-defense system of systems for surface warship formation based on danger theory

Aiming at the problem on cooperative air-defense of surface warship formation, this paper maps the cooperative airdefense system of systems （SoS） for surface warship formation （CASoSSWF） to the biological immune system （BIS） according to the similarity of the defense mechanism and characteristics between the CASoSSWF and the BIS, and then designs the models of components and the architecture for a monitoring agent, a regulating agent, a killer agent, a pre-warning agent and a communicating agent by making use of the theories and methods of the artificial immune system, the multi-agent system （MAS）, the vaccine and the danger theory （DT）. Moreover a new immune multi-agent model using vaccine based on DT （IMMUVBDT） for the cooperative air-defense SoS is advanced. The immune response and immune mechanism of the CASoSSWF are analyzed. The model has a capability of memory, evolution, commendable dynamic environment adaptability and self-learning, and embodies adequately the cooperative air-defense mechanism for the CASoSSWF. Therefore it shows a novel idea for the CASoSSWF which can provide conception models for a surface warship formation operation simulation system.

Multi-agent Based Hierarchy Simulation Models of Carrier-based Aircraft Catapult Launch

With the aid of multi-agent based modeling approach to complex systems, the hierarchy simulation models of carrier-based aircraft catapult launch are developed. Ocean, carrier, aircraft, and atmosphere are treated as aggregation agents, the detailed components like catapult, landing gears, and disturbances are considered as meta-agents, which belong to their aggregation agent. Thus, the model with two layers is formed i.e. the aggregation agent layer and the meta-agent layer. The information communication among all agents is described. The meta-agents within one aggregation agent communicate with each other directly by information sharing, but the meta-agents, which belong to different aggregation agents exchange their information through the aggregation layer first, and then perceive it from the sharing environment, that is the aggregation agent. Thus, not only the hierarchy model is built, but also the environment perceived by each agent is specified. Meanwhile, the problem of balancing the independency of agent and the resource consumption brought by real-time communication within multi-agent system （MAS） is resolved. Each agent involved in carrier-based aircraft catapult launch is depicted, with considering the interaction within disturbed atmospheric environment and multiple motion bodies including carrier, aircraft, and landing gears. The models of reactive agents among them are derived based on tensors, and the perceived messages and inner frameworks of each agent are characterized. Finally, some results of a simulation instance are given. The simulation and modeling of dynamic system based on multi-agent system is of benefit to express physical concepts and logical hierarchy clearly and precisely. The system model can easily draw in kinds of other agents to achieve a precise simulation of more complex system. This modeling technique makes the complex integral dynamic equations of multibodies decompose into parallel operations of single agent, and it is convenient to expand, maintain, and reuse the program codes.

Distributed collaborative complete coverage path planning based on hybrid strategy

Collaborative coverage path planning(CCPP) refers to obtaining the shortest paths passing over all places except obstacles in a certain area or space. A multi-unmanned aerial vehicle(UAV) collaborative CCPP algorithm is proposed for the urban rescue search or military search in outdoor environment.Due to flexible control of small UAVs, it can be considered that all UAVs fly at the same altitude, that is, they perform search tasks on a two-dimensional plane. Based on the agents’ motion characteristics and environmental information, a mathematical model of CCPP problem is established. The minimum time for UAVs to complete the CCPP is the objective function, and complete coverage constraint, no-fly constraint, collision avoidance constraint, and communication constraint are considered. Four motion strategies and two communication strategies are designed. Then a distributed CCPP algorithm is designed based on hybrid strategies. Simulation results compared with patternbased genetic algorithm(PBGA) and random search method show that the proposed method has stronger real-time performance and better scalability and can complete the complete CCPP task more efficiently and stably.

Robust adaptive leaderless consensus of unknown non-minimum phase linear multi-agent systems subject to disturbances and/or unmodeled dynamics

This article investigates the problem of robust adaptive leaderless consensus for heterogeneous uncertain nonminimumphase linear multi-agent systems over directed communication graphs. Each agent is assumed tobe of unknown nominal dynamics and also subject to external disturbances and/or unmodeled dynamics. Anovel distributed robust adaptive control strategy is proposed. It is shown that the robust adaptive leaderlessconsensus problem is solved with the proposed control strategy under some sufficient conditions. Two examplesare provided to demonstrate the efficacy of the proposed control strategy.

Policy driven and multi-agent based fault tolerance for Web services

This paper proposes a policy driven and multi-agent based model to enhance the fault tolerance and recovery capabilities of Web services in distributed environment. The evaluation function of fault specifications and the corresponding handling mechanisms of the services are both defined in policies, which are expressed in XML. During the implementation of the services,the occurrences of faults are monitored by the service monitor agent through the local knowledge on the faults. Such local knowledge is dynamically generated by the service policy agent through querying and parsing the service policies from the service policies repository. When the fault occurs, the service process agent will focus on the process of fault handling and service recovery, which will be directed with the actions defined in the policies upon the specific conditions. Such a policy driven and multi-agent based fault handling approach can address the issues of flexibility, automation and availability.

Market based framework for multiple AUVs cooperation

A“Market” based framework for multiple AUVs team is introduced in this paper.It is a distributed meta-level task allocation framwork. The formulation and the basic concepts of the “Market” such as “goods” and “price” are discussed first, then the basic algorithm of the “auction”. The loosely coupled v-MDTSP tasks are considered as an example of the task allocation mission. A multiple AUV team controller and a detailed algorithm are developed for such applications. The simulation results show that the controller has the advantages such as robustness and low complexity and it can achieve better optimization results than the classical central controller (such as GA) in some tasks. And the comparison of two different local solvers also implies that we should get the reasonable task allocation even not using the high quality algorithm, which can considerably decrease the cooperation computation.

Research on Cognitive Cooperation in Multi-Agent Systems

The accomplishment of a complex problem usually involves cooperation between participators with different knowledge background concerned. This paper identifies interdependency between different sub problems （through problem decomposition） as the major factor that influences cooperative relations in multi-Agent systems, based on which we propose an efficient means to measure cooperation coefficient （degree） between different Agents. Then cognitive cooperation between Agents is analyzed which aims at collecting the wisdom of the cognitive community for a systematic solution to the overall problem.

STABILITY ANALYSIS AND COOPERATIVE CONTROL OF DISTRIBUTED MULTI-AGENT SYSTEM WITH SAMPLED COMMUNICATION

The cooperative control and stability analysis problems for the multi-agent system with sampled com- munication are investigated. Distributed state feedback controllers are adopted for the cooperation of networked agents. A theorem in the form of linear matrix inequalities（LMI） is derived to analyze the system stability. An- other theorem in the form of optimization problem subject to LMI constraints is proposed to design the controller, and then the algorithm is presented. The simulation results verify the validity and the effectiveness of the pro- posed approach.

Circular Formation Control with Collision Avoidance Based on Probabilistic Position

In this paper,we study the circular formation problem for the second-order multi-agent systems in a plane,in which the agents maintain a circular formation based on a probabilistic position.A distributed hybrid control protocol based on a probabilistic position is designed to achieve circular formation stabilization and consensus.In the current framework,the mobile agents follow the following rules:1)the agent must follow a circular trajectory;2)all the agents in the same circular trajectory must have the same direction.The formation control objective includes two parts:1)drive all the agents to the circular formation;2)avoid a collision.Based on Lyapunov methods,convergence and stability of the proposed circular formation protocol are provided.Due to limitations in collision avoidance,we extend the results to LaSalle’s invariance principle.Some theoretical examples and numerical simulations show the effectiveness of the proposed scheme.

Cooperative Subcarrier Sensing Using Antenna Diversity Based Weighted Virtual Sub Clustering

The idea of cooperation and the clustering amongst cognitive radios(CRs) has recently been focus of attention of research community, owing to its potential to improve performance of spectrum sensing(SS) schemes. This focus has led to the paradigm of cluster based cooperative spectrum sensing(CBCSS). In perspective of high date rate 4th generation wireless systems, which are characterized by orthogonal frequency division multiplexing(OFDM) and spatial diversity, there is a need to devise effective SS strategies. A novel CBCSS scheme is proposed for OFDM subcarrier detection in order to enable the non-contiguous OFDM(NC-OFDM) at the physical layer of CRs for efficient utilization of spectrum holes. Proposed scheme is based on the energy detection in MIMO CR network, using equal gain combiner as diversity combining technique, hard combining(AND, OR and Majority) rule as data fusion technique and antenna diversity based weighted clustering as virtual sub clustering algorithm. Results of proposed CBCSS are compared with conventional CBCSS scheme for AND, OR and Majority data fusion rules. Moreover the effects of antenna diversity, cooperation and cooperating clusters are also discussed.

A 3D Geometry-Based Scattering Model for Vehicleto-Vehicle Wideband MIMO Relay-Based Cooperative Channels

In this paper, a three-dimensional(3D) geometry- based stochastic scattering model(GBSSM) for wideband multi-input multi-output(MIMO) vehicle-to-vehicle(V2V) relay-based cooperative fading channel based on geometrical three-cylinder is proposed. Non-line-of-sight(NLOS) propagation condition is assumed in amplify-and-forward(AF) cooperative networks from the source mobile station(S) to the destination mobile station(D) via the mobile relay station(R). We extend the proposed narrowband model to wideband and also introduce the carrier frequency and bandwidth into the model. To avoid complicated procedure in deriving the analytical expressions of the channel parameters and functions, the channel is realized first. By using the realized channel matrix, the channel properties are further investigated.

Hiatt-Neu-Cooper神经发育综合征一例

Hiatt-Neu-Cooper神经发育综合征(HINCONS)是一种与7p14上RALA的杂合突变密切相关、以智力发育障碍或全面性发育落后为突出表现的罕见神经发育遗传病,目前尚无有效治疗手段。本文报道1例HINCONS患儿,主要表现为全面性发育落后,特别是语言和运动发育落后、全身肌张力减低、身材矮小、特殊面容及先天性室间隔缺损,实验室生化检查未见明显异常,头颅MRI提示透明隔间隙、穹窿间隙增宽,家系全外显子组测序提示患儿携带RALA基因c.475(exon4)A>G(p.K159E)变异,为新发变异。现结合文献报道的12例HINCONS患者临床特征进行回顾分析,以期为HINCONS的临床诊断与遗传咨询提供参考依据。

Cooperative Multi-Agent Reinforcement Learning with Constraint-Reduced DCOP

Cooperative multi-agent reinforcement learning（ MARL） is an important topic in the field of artificial intelligence,in which distributed constraint optimization（ DCOP） algorithms have been widely used to coordinate the actions of multiple agents. However,dense communication among agents affects the practicability of DCOP algorithms. In this paper,we propose a novel DCOP algorithm dealing with the previous DCOP algorithms＇ communication problem by reducing constraints.The contributions of this paper are primarily threefold：（1） It is proved that removing constraints can effectively reduce the communication burden of DCOP algorithms.（2） An criterion is provided to identify insignificant constraints whose elimination doesn＇t have a great impact on the performance of the whole system.（3） A constraint-reduced DCOP algorithm is proposed by adopting a variant of spectral clustering algorithm to detect and eliminate the insignificant constraints. Our algorithm reduces the communication burdern of the benchmark DCOP algorithm while keeping its overall performance unaffected. The performance of constraint-reduced DCOP algorithm is evaluated on four configurations of cooperative sensor networks. The effectiveness of communication reduction is also verified by comparisons between the constraint-reduced DCOP and the benchmark DCOP.

Cooperative output tracking of multi-agent systems under finite time

The cooperative output tracking problem of multi-agent systems in finite time is considered.In order to enable the agents to quickly track and converge to external system within a finite time,a novel distributed output feedback control strategy based on the finite-time state observer is designed.This distributed finite-time observer can not only solve cooperative output tracking problems when the agents can not get external system signal,but also make the systems have a faster convergence and a good robustness.The stability of the system in finite time is proved based on Lyapunov function.Numerical simulations results have been provided to demonstrate the effectiveness of the proposed protocol.