Regional Multi-Agent Cooperative Reinforcement Learning for City-Level Traffic Grid Signal Control

This article studies the effective traffic signal control problem of multiple intersections in a city-level traffic system.A novel regional multi-agent cooperative reinforcement learning algorithm called RegionSTLight is proposed to improve the traffic efficiency.Firstly a regional multi-agent Q-learning framework is proposed,which can equivalently decompose the global Q value of the traffic system into the local values of several regions Based on the framework and the idea of human-machine cooperation,a dynamic zoning method is designed to divide the traffic network into several strong-coupled regions according to realtime traffic flow densities.In order to achieve better cooperation inside each region,a lightweight spatio-temporal fusion feature extraction network is designed.The experiments in synthetic real-world and city-level scenarios show that the proposed RegionS TLight converges more quickly,is more stable,and obtains better asymptotic performance compared to state-of-theart models.

Event-triggered robust cooperative output regulation for a class of linear multi-agent systems with an unknown exosystem

This paper investigates the robust cooperative output regulation problem for a class of heterogeneousuncertain linear multi-agent systems with an unknown exosystem via event-triggered control (ETC). By utilizingthe internal model approach and the adaptive control technique, a distributed adaptive internal model isconstructed for each agent. Then, based on this internal model, a fully distributed ETC strategy composed ofa distributed event-triggered adaptive output feedback control law and a distributed dynamic event-triggeringmechanism is proposed, in which each agent updates its control input at its own triggering time instants. It isshown that under the proposed ETC strategy, the robust cooperative output regulation problem can be solvedwithout requiring either the global information associated with the communication topology or the bounds ofthe uncertain or unknown parameters in each agent and the exosystem. A numerical example is provided toillustrate the effectiveness of the proposed control strategy.

Finite-time Prescribed Performance Time-Varying Formation Control for Second-Order Multi-Agent Systems With Non-Strict Feedback Based on a Neural Network Observer

This paper studies the problem of time-varying formation control with finite-time prescribed performance for nonstrict feedback second-order multi-agent systems with unmeasured states and unknown nonlinearities.To eliminate nonlinearities,neural networks are applied to approximate the inherent dynamics of the system.In addition,due to the limitations of the actual working conditions,each follower agent can only obtain the locally measurable partial state information of the leader agent.To address this problem,a neural network state observer based on the leader state information is designed.Then,a finite-time prescribed performance adaptive output feedback control strategy is proposed by restricting the sliding mode surface to a prescribed region,which ensures that the closed-loop system has practical finite-time stability and that formation errors of the multi-agent systems converge to the prescribed performance bound in finite time.Finally,a numerical simulation is provided to demonstrate the practicality and effectiveness of the developed algorithm.

A PI+R Control Scheme Based on Multi-Agent Systems for Economic Dispatch in Isolated BESSs

Battery energy storage systems(BESSs)are widely used in smart grids.However,power consumed by inner impedance and the capacity degradation of each battery unit become particularly severe,which has resulted in an increase in operating costs.The general economic dispatch(ED)algorithm based on marginal cost(MC)consensus is usually a proportional(P)controller,which encounters the defects of slow convergence speed and low control accuracy.In order to solve the distributed ED problem of the isolated BESS network with excellent dynamic and steady-state performance,we attempt to design a proportional integral(PI)controller with a reset mechanism(PI+R)to asymptotically promote MC consensus and total power mismatch towards 0 in this paper.To be frank,the integral term in the PI controller is reset to 0 at an appropriate time when the proportional term undergoes a zero crossing,which accelerates convergence,improves control accuracy,and avoids overshoot.The eigenvalues of the system under a PI+R controller is well analyzed,ensuring the regularity of the system and enabling the reset mechanism.To ensure supply and demand balance within the isolated BESSs,a centralized reset mechanism is introduced,so that the controller is distributed in a flow set and centralized in a jump set.To cope with Zeno behavior and input delay,a dwell time that the system resides in a flow set is given.Based on this,the system with input delays can be reduced to a time-delay free system.Considering the capacity limitation of the battery,a modified MC scheme with PI+R controller is designed.The correctness of the designed scheme is verified through relevant simulations.

Targeted multi-agent communication algorithm based on state control

As an important mechanism in multi-agent interaction,communication can make agents form complex team relationships rather than constitute a simple set of multiple independent agents.However,the existing communication schemes can bring much timing redundancy and irrelevant messages,which seriously affects their practical application.To solve this problem,this paper proposes a targeted multiagent communication algorithm based on state control(SCTC).The SCTC uses a gating mechanism based on state control to reduce the timing redundancy of communication between agents and determines the interaction relationship between agents and the importance weight of a communication message through a series connection of hard-and self-attention mechanisms,realizing targeted communication message processing.In addition,by minimizing the difference between the fusion message generated from a real communication message of each agent and a fusion message generated from the buffered message,the correctness of the final action choice of the agent is ensured.Our evaluation using a challenging set of Star Craft II benchmarks indicates that the SCTC can significantly improve the learning performance and reduce the communication overhead between agents,thus ensuring better cooperation between agents.

Safety-Constrained Multi-Agent Reinforcement Learning for Power Quality Control in Distributed Renewable Energy Networks

This paper examines the difficulties of managing distributed power systems,notably due to the increasing use of renewable energy sources,and focuses on voltage control challenges exacerbated by their variable nature in modern power grids.To tackle the unique challenges of voltage control in distributed renewable energy networks,researchers are increasingly turning towards multi-agent reinforcement learning(MARL).However,MARL raises safety concerns due to the unpredictability in agent actions during their exploration phase.This unpredictability can lead to unsafe control measures.To mitigate these safety concerns in MARL-based voltage control,our study introduces a novel approach:Safety-ConstrainedMulti-Agent Reinforcement Learning(SC-MARL).This approach incorporates a specialized safety constraint module specifically designed for voltage control within the MARL framework.This module ensures that the MARL agents carry out voltage control actions safely.The experiments demonstrate that,in the 33-buses,141-buses,and 322-buses power systems,employing SC-MARL for voltage control resulted in a reduction of the Voltage Out of Control Rate(%V.out)from0.43,0.24,and 2.95 to 0,0.01,and 0.03,respectively.Additionally,the Reactive Power Loss(Q loss)decreased from 0.095,0.547,and 0.017 to 0.062,0.452,and 0.016 in the corresponding systems.

Neural Dynamics for Cooperative Motion Control of Omnidirectional Mobile Manipulators in the Presence of Noises: A Distributed Approach

This paper presents a distributed scheme with limited communications, aiming to achieve cooperative motion control for multiple omnidirectional mobile manipulators(MOMMs).The proposed scheme extends the existing single-agent motion control to cater to scenarios involving the cooperative operation of MOMMs. Specifically, squeeze-free cooperative load transportation is achieved for the end-effectors of MOMMs by incorporating cooperative repetitive motion planning(CRMP), while guiding each individual to desired poses. Then, the distributed scheme is formulated as a time-varying quadratic programming(QP) and solved online utilizing a noise-tolerant zeroing neural network(NTZNN). Theoretical analysis shows that the NTZNN model converges globally to the optimal solution of QP in the presence of noise. Finally, the effectiveness of the control design is demonstrated by numerical simulations and physical platform experiments.

STABILITY ANALYSIS AND COOPERATIVE CONTROL OF DISTRIBUTED MULTI-AGENT SYSTEM WITH SAMPLED COMMUNICATION

The cooperative control and stability analysis problems for the multi-agent system with sampled com- munication are investigated. Distributed state feedback controllers are adopted for the cooperation of networked agents. A theorem in the form of linear matrix inequalities（LMI） is derived to analyze the system stability. An- other theorem in the form of optimization problem subject to LMI constraints is proposed to design the controller, and then the algorithm is presented. The simulation results verify the validity and the effectiveness of the pro- posed approach.

Cooperative multi-target hunting by unmanned surface vehicles based on multi-agent reinforcement learning

To solve the problem of multi-target hunting by an unmanned surface vehicle(USV)fleet,a hunting algorithm based on multi-agent reinforcement learning is proposed.Firstly,the hunting environment and kinematic model without boundary constraints are built,and the criteria for successful target capture are given.Then,the cooperative hunting problem of a USV fleet is modeled as a decentralized partially observable Markov decision process(Dec-POMDP),and a distributed partially observable multitarget hunting Proximal Policy Optimization(DPOMH-PPO)algorithm applicable to USVs is proposed.In addition,an observation model,a reward function and the action space applicable to multi-target hunting tasks are designed.To deal with the dynamic change of observational feature dimension input by partially observable systems,a feature embedding block is proposed.By combining the two feature compression methods of column-wise max pooling(CMP)and column-wise average-pooling(CAP),observational feature encoding is established.Finally,the centralized training and decentralized execution framework is adopted to complete the training of hunting strategy.Each USV in the fleet shares the same policy and perform actions independently.Simulation experiments have verified the effectiveness of the DPOMH-PPO algorithm in the test scenarios with different numbers of USVs.Moreover,the advantages of the proposed model are comprehensively analyzed from the aspects of algorithm performance,migration effect in task scenarios and self-organization capability after being damaged,the potential deployment and application of DPOMH-PPO in the real environment is verified.

Distributed cooperative control of autonomous multi-agent UAV systems using smooth control

This paper addresses the cooperative control problem of multiple unmanned aerial vehicles(multi-UAV)systems.First,a new distributed consensus algorithm for second-order nonlinear multi-agent systems(MAS)is formulated under the leader-following approach.The algorithm provides smooth input signals to the agents’control channels,which avoids the chattering effect generated by the conventional sliding mode-based control protocols.Second,a new formation control scheme is developed by integrating smooth distributed consensus control protocols into the geometric pattern model to achieve three-dimensional formation tracking.The Lyapunov theory is used to prove the stability and convergence of both distributed consensus and formation controllers.The effectiveness of the proposed algorithms is demonstrated through simulation results.

Cooperative Multi-Agent Control for Autonomous Ship Towing Under Environmental Disturbances

Among the promising application of autonomous surface vessels(ASVs)is the utilization of multiple autonomous tugs for manipulating a floating object such as an oil platform,a broken ship,or a ship in port areas.Considering the real conditions and operations of maritime practice,this paper proposes a multi-agent control algorithm to manipulate a ship to a desired position with a desired heading and velocity under the environmental disturbances.The control architecture consists of a supervisory controller in the higher layer and tug controllers in the lower layer.The supervisory controller allocates the towing forces and angles between the tugs and the ship by minimizing the error in the position and velocity of the ship.The weight coefficients in the cost function are designed to be adaptive to guarantee that the towing system functions well under environmental disturbances,and to enhance the efficiency of the towing system.The tug controller provides the forces to tow the ship and tracks the reference trajectory that is computed online based on the towing angles calculated by the supervisory controller.Simulation results show that the proposed algorithm can make the two autonomous tugs cooperatively tow a ship to a desired position with a desired heading and velocity under the(even harsh)environmental disturbances.

Robust adaptive leaderless consensus of unknown non-minimum phase linear multi-agent systems subject to disturbances and/or unmodeled dynamics

This article investigates the problem of robust adaptive leaderless consensus for heterogeneous uncertain nonminimumphase linear multi-agent systems over directed communication graphs. Each agent is assumed tobe of unknown nominal dynamics and also subject to external disturbances and/or unmodeled dynamics. Anovel distributed robust adaptive control strategy is proposed. It is shown that the robust adaptive leaderlessconsensus problem is solved with the proposed control strategy under some sufficient conditions. Two examplesare provided to demonstrate the efficacy of the proposed control strategy.

Robust Consensus Tracking Control of Uncertain Multi-Agent Systems With Local Disturbance Rejection

In this paper,a new distributed consensus tracking protocol incorporating local disturbance rejection is devised for a multi-agent system with heterogeneous dynamic uncertainties and disturbances over a directed graph.It is of two-degree-of-freedom nature.Specifically,a robust distributed controller is designed for consensus tracking,while a local disturbance estimator is designed for each agent without requiring the input channel information of disturbances.The condition for asymptotic disturbance rejection is derived.Moreover,even when the disturbance model is not exactly known,the developed method also provides good disturbance-rejection performance.Then,a robust stabilization condition with less conservativeness is derived for the whole multi-agent system.Further,a design algorithm is given.Finally,comparisons with the conventional one-degree-of-freedombased distributed disturbance-rejection method for mismatched disturbances and the distributed extended-state observer for matched disturbances validate the developed method.

Multi-UAV cooperative maneuver decision-making for pursuitevasion using improved MADRL

Aiming at the problem of multi-UAV pursuit-evasion confrontation, a UAV cooperative maneuver method based on an improved multi-agent deep reinforcement learning(MADRL) is proposed. In this method, an improved Comm Net network based on a communication mechanism is introduced into a deep reinforcement learning algorithm to solve the multi-agent problem. A layer of gated recurrent unit(GRU) is added to the actor-network structure to remember historical environmental states. Subsequently,another GRU is designed as a communication channel in the Comm Net core network layer to refine communication information between UAVs. Finally, the simulation results of the algorithm in two sets of scenarios are given, and the results show that the method has good effectiveness and applicability.

Cooperative Rate Splitting Transmit Design for Full-Duplex-Enabled Multiple Multicast Communication Systems

This paper examines the performance of Full-Duplex Cooperative Rate Splitting(FD-CRS)with Simultaneous Wireless Information and Power Transfer(SWIPT)support in Multiple Input Single Output(MISO)networks.In a Rate Splitting Multiple Access(RSMA)multicast system with two local users and one remote user,the common data stream contains the needs of all users,and all users can decode the common data stream.Therefore,each user can receive some information that other users need,and local users with better channel conditions can use this information to further enhance the reception reliability and data rate of users with poor channel quality.Even using Cell-Center-Users(CCUs)as a cooperative relay to assist the transmission of common data can improve the average system speed.To maximize the minimum achievable rate,we optimize the beamforming vector of Base Station(BS),the common streamsplitting vector,the cooperative distributed beamvector and the strong user transmission power under the power budget constraints of BS and relay devices and the service quality requirements constraints of users.Since the whole problem is not convex,we cannot solve it directly.Therefore,we propose a low complexity algorithm based on Successive Convex Approximation(SCA)technology to find the optimal solution to the problemunder consideration.The simulation results show that FD C-RSMA has better gain andmore powerful than FD C-NOMA,HD C-RSMA,RSMA and NOMA.

An Optimal Control-Based Distributed Reinforcement Learning Framework for A Class of Non-Convex Objective Functionals of the Multi-Agent Network

This paper studies a novel distributed optimization problem that aims to minimize the sum of the non-convex objective functionals of the multi-agent network under privacy protection, which means that the local objective of each agent is unknown to others. The above problem involves complexity simultaneously in the time and space aspects. Yet existing works about distributed optimization mainly consider privacy protection in the space aspect where the decision variable is a vector with finite dimensions. In contrast, when the time aspect is considered in this paper, the decision variable is a continuous function concerning time. Hence, the minimization of the overall functional belongs to the calculus of variations. Traditional works usually aim to seek the optimal decision function. Due to privacy protection and non-convexity, the Euler-Lagrange equation of the proposed problem is a complicated partial differential equation.Hence, we seek the optimal decision derivative function rather than the decision function. This manner can be regarded as seeking the control input for an optimal control problem, for which we propose a centralized reinforcement learning(RL) framework. In the space aspect, we further present a distributed reinforcement learning framework to deal with the impact of privacy protection. Finally, rigorous theoretical analysis and simulation validate the effectiveness of our framework.

Multi-agent based modeling for order distribution cooperative negotiation in supply chains

With the new characteristics of global cooperation in supply chains being synthetically considered,a hybrid model to the cooperative negotiation process for the order distribution in supply chain is mainly studied.After reviewing and analyzing some main domestic and overseas processes in cooperative negotiation modeling in supply chain,some problems are subsequently pointed out.For example,the traditional simple multi-agent system(MAS)frameworks which have some limitations,are not suitable for solving modeling complex systems.To solve these problems,thinking with the aid of the multi-agent structure and complex system modeling,the manufacturing supply chain is taken as an example,and a time Petri net production model is adopted to decompose the materials.And then a cooperative negotiation model for the order distribution in supply chain is constructed based on combining multi-agent techniques with time Petri net modeling.The simulation results reveal that the above model helps solve the problems of cooperative negotiation in supply chains.

Practical bipartite consensus for multi-agent systems:A barrier function-based adaptive sliding-mode control approach

This paper is concerned with bipartite consensus tracking for multi-agent systems with unknown disturbances.A barrier function-based adaptive sliding-mode control(SMC)approach is proposed such that the bipartite steady-state error is converged to a predefined region of zero in finite time.Specifically,based on an error auxiliary taking neighboring antagonistic interactions into account,an SMC law is designed with an adaptive gain.The gain can switch to a positive semi-definite barrier function to ensure that the error auxiliary is constrained to a predefined neighborhood of zero,which in turn guarantees practical bipartite consensus tracking.A distinguished feature of the proposed controller is its independence on the bound of disturbances,while the input chattering phenomenon is alleviated.Finally,a numerical example is provided to verify the effectiveness of the proposed controller.

Lyapunov-Based Output Containment Control of Heterogeneous Multi-Agent Systems With Markovian Switching Topologies and Distributed Delays

This paper considers the mean square output containment control problem for heterogeneous multi-agent systems(MASs)with randomly switching topologies and nonuniform distributed delays.By modeling the switching topologies as a continuous-time Markov process and taking the distributed delays into consideration,a novel distributed containment observer is proposed to estimate the convex hull spanned by the leaders'states.A novel distributed output feedback containment controller is then designed without using the prior knowledge of distributed delays.By constructing a novel switching Lyapunov functional,the output containment control problem is then solved in the sense of mean square under an easily-verifiable sufficient condition.Finally,two numerical examples are given to show the effectiveness of the proposed controller.

Power Optimization Cooperative Control Strategy for Flexible Fast Interconnection Device with Energy Storage

With the wide application of renewable energy power generation technology,the distribution network presents the characteristics of multi-source and complex structure.There are potential risks in the stability of power system,and the problem of power quality is becoming more and more serious.This paper studies and proposes a power optimization cooperative control strategy for flexible fast interconnection device with energy storage,which combines the flexible interconnection technology with the energy storage device.The primary technology is to regulate the active and reactive power of the converter.By comparing the actual power value of the converter with the reference value,the proportional integral(PI)controller is used for correction,and the current components of d and q axes are obtained and input to the converter as the reference value of the current inner loop.The control strategy in this paper can realize power mutual aid between feeders,and at the same time,the energy storage device can provide or absorb a certain amount of power for feeders,so that the power grid can realize stable operation in a certain range.