An Optimal Control-Based Distributed Reinforcement Learning Framework for A Class of Non-Convex Objective Functionals of the Multi-Agent Network

This paper studies a novel distributed optimization problem that aims to minimize the sum of the non-convex objective functionals of the multi-agent network under privacy protection, which means that the local objective of each agent is unknown to others. The above problem involves complexity simultaneously in the time and space aspects. Yet existing works about distributed optimization mainly consider privacy protection in the space aspect where the decision variable is a vector with finite dimensions. In contrast, when the time aspect is considered in this paper, the decision variable is a continuous function concerning time. Hence, the minimization of the overall functional belongs to the calculus of variations. Traditional works usually aim to seek the optimal decision function. Due to privacy protection and non-convexity, the Euler-Lagrange equation of the proposed problem is a complicated partial differential equation.Hence, we seek the optimal decision derivative function rather than the decision function. This manner can be regarded as seeking the control input for an optimal control problem, for which we propose a centralized reinforcement learning(RL) framework. In the space aspect, we further present a distributed reinforcement learning framework to deal with the impact of privacy protection. Finally, rigorous theoretical analysis and simulation validate the effectiveness of our framework.

A Cooperative Security Monitoring Mechanism Aided by Optimal Multiple Slave Cluster Heads for UASNs

As each cluster head(CH)sensor node is used to aggregate,fuse,and forward data from different sensor nodes in an underwater acoustic sensor network(UASN),guaranteeing the data security in a CH is very critical.In this paper,a cooperative security monitoring mechanism aided by multiple slave cluster heads(SCHs)is proposed to keep track of the data security of a CH.By designing a low complexity“equilateral triangle algorithm(ETA)”,the optimal SCHs(named as ETA-based multiple SCHs)are selected from the candidate SCHs so as to improve the dispersion and coverage of SCHs and achieve largescale data security monitoring.In addition,by analyzing the entire monitoring process,the close form expression of the probability of the failure attack identification for the SCHs with respect to the probability of attack launched by ordinary nodes is deduced.The simulation results show that the proposed optimal ETA-based multiple SCH cooperation scheme has lower probability of the failure attack identification than that of the existing schemes.In addition,the numerical simulation results are consistent with the theoretical analysis results,thus verifying the effectiveness of the proposed scheme.

Distributed Stochastic Optimization with Compression for Non-Strongly Convex Objectives

We are investigating the distributed optimization problem,where a network of nodes works together to minimize a global objective that is a finite sum of their stored local functions.Since nodes exchange optimization parameters through the wireless network,large-scale training models can create communication bottlenecks,resulting in slower training times.To address this issue,CHOCO-SGD was proposed,which allows compressing information with arbitrary precision without reducing the convergence rate for strongly convex objective functions.Nevertheless,most convex functions are not strongly convex(such as logistic regression or Lasso),which raises the question of whether this algorithm can be applied to non-strongly convex functions.In this paper,we provide the first theoretical analysis of the convergence rate of CHOCO-SGD on non-strongly convex objectives.We derive a sufficient condition,which limits the fidelity of compression,to guarantee convergence.Moreover,our analysis demonstrates that within the fidelity threshold,this algorithm can significantly reduce transmission burden while maintaining the same convergence rate order as its no-compression equivalent.Numerical experiments further validate the theoretical findings by demonstrating that CHOCO-SGD improves communication efficiency and keeps the same convergence rate order simultaneously.And experiments also show that the algorithm fails to converge with low compression fidelity and in time-varying topologies.Overall,our study offers valuable insights into the potential applicability of CHOCO-SGD for non-strongly convex objectives.Additionally,we provide practical guidelines for researchers seeking to utilize this algorithm in real-world scenarios.

Fully Distributed Learning for Deep Random Vector Functional-Link Networks

In the contemporary era, the proliferation of information technology has led to an unprecedented surge in data generation, with this data being dispersed across a multitude of mobile devices. Facing these situations and the training of deep learning model that needs great computing power support, the distributed algorithm that can carry out multi-party joint modeling has attracted everyone’s attention. The distributed training mode relieves the huge pressure of centralized model on computer computing power and communication. However, most distributed algorithms currently work in a master-slave mode, often including a central server for coordination, which to some extent will cause communication pressure, data leakage, privacy violations and other issues. To solve these problems, a decentralized fully distributed algorithm based on deep random weight neural network is proposed. The algorithm decomposes the original objective function into several sub-problems under consistency constraints, combines the decentralized average consensus (DAC) and alternating direction method of multipliers (ADMM), and achieves the goal of joint modeling and training through local calculation and communication of each node. Finally, we compare the proposed decentralized algorithm with several centralized deep neural networks with random weights, and experimental results demonstrate the effectiveness of the proposed algorithm.

Distributed Periodic Event-Triggered Optimal Control of DC Microgrids Based on Virtual Incremental Cost

This article presents a distributed periodic eventtriggered(PET)optimal control scheme to achieve generation cost minimization and average bus voltage regulation in DC microgrids.In order to accommodate the generation constraints of the distributed generators(DGs),a virtual incremental cost is firstly designed,based on which an optimality condition is derived to facilitate the control design.To meet the discrete-time(DT)nature of modern control systems,the optimal controller is directly developed in the DT domain.Afterward,to reduce the communication requirement among the controllers,a distributed event-triggered mechanism is introduced for the DT optimal controller.The event-triggered condition is detected periodically and therefore naturally avoids the Zeno phenomenon.The closed-loop system stability is proved by the Lyapunov synthesis for switched systems.The generation cost minimization and average bus voltage regulation are obtained at the equilibrium point.Finally,switch-level microgrid simulations validate the performance of the proposed optimal controller.

A Distributed Optimal Scheme Based on Local QoS for Web Service Composition

The goal of web service composition is to choose an optimal scheme according to Quantity of Service （QoS） which selects instances in a distributed network. The networks are clustered with some web services such as ontologies, algorithms and rule engines with similar function and interfaces. In this scheme, web services acted as candidate service construct a distributed model which can＇t obtain the global services＇ information. The model is utilized to choose instances according to local QoS information in the progress of service composition. Some QoS matrixes are used to record and compare the instance paths and then choose a better one. Simulation result has proven that our ~pproach has a tradeoff between efficiency and ~quality.

Cycle Flow Formulation of Optimal Network Flow Problems and Respective Distributed Solutions

In this paper, we use the cycle basis from graph theory to reduce the size of the decision variable space of optimal network flow problems by eliminating the aggregated flow conservation constraint. We use a minimum cost flow problem and an optimal power flow problem with generation and storage at the nodes to demonstrate our decision variable reduction method.The main advantage of the proposed technique is that it retains the natural sparse/decomposable structure of network flow problems. As such, the reformulated problems are still amenable to distributed solutions. We demonstrate this by proposing a distributed alternating direction method of multipliers(ADMM)solution for a minimum cost flow problem. We also show that the communication cost of the distributed ADMM algorithm for our proposed cycle-based formulation of the minimum cost flow problem is lower than that of a distributed ADMM algorithm for the original arc-based formulation.

Spatial batch optimal design based on self-learning Gaussian process models for LPCVD processes

Low pressure chemical vapor deposition(LPCVD) is one of the most important processes during semiconductor manufacturing.However,the spatial distribution of internal temperature and extremely few samples makes it hard to build a good-quality model of this batch process.Besides,due to the properties of this process,the reliability of the model must be taken into consideration when optimizing the MVs.In this work,an optimal design strategy based on the self-learning Gaussian process model(GPM) is proposed to control this kind of spatial batch process.The GPM is utilized as the internal model to predict the thicknesses of thin films on all spatial-distributed wafers using the limited data.Unlike the conventional model based design,the uncertainties of predictions provided by GPM are taken into consideration to guide the optimal design of manipulated variables so that the designing can be more prudent Besides,the GPM is also actively enhanced using as little data as possible based on the predictive uncertainties.The effectiveness of the proposed strategy is successfully demonstrated in an LPCVD process.

Optimal distribution of reliability for a large network based on connectivity

It is a non-polynomial complexity problem to calculate connectivity of the complex network. When the system reliability cannot be expressed as a function of element reliability, we have to apply some heuristic methods for optimization based on connectivity of the network. The calculation structure of connectivity of complex network is analyzed in the paper. The coefficient matrixes of Taylor second order expansion of the system connectivity is generated based on the calculation structure of connectivity of complex network. An optimal schedule is achieved based on genetic algorithms （GA）. Fitness of seeds is calculated using the Taylor expansion function of system connectivity. Precise connectivity of the optimal schedule and the Taylor expansion function of system connectivity can be achieved by the approved Minty method or the recursive decomposition algorithm. When error between approximate connectivity and the precise value exceeds the assigned value, the optimization process is continued using GA, and the Taylor function of system connectivity needs to be renewed. The optimization process is called iterative GA. Iterative GA can be used in the large network for optimal reliability attribution. One temporary optimal result will be generated every time in the iteration process. These temporary optimal results approach the real optimal results. They can be regarded as a group of approximate optimal results useful in the real project.

DISTRIBUTED OPTIMAL LOCAL DOUBLE LOOP NETWORK

A distibuted optimal local double loop(DOLDL) network is presented. Emphasis is laid on the topology and distributed routing algorithms for the DOLDL. On the basis of building an abstract model, a set of definitions and theorems are described and proved. An algorithm which can optimize the double loop networks is presented. The optimal values of the topologic parameters for the DOLDL have been obtained by the algorithm, and these numerical results are analyzed. The study shows that the bounds of the optimal diameter (d) and average hop distance (a) for this class of networks are [square-root 3N -2] less-than-or-equal-to d less-than-or-equal-to [square-root 3N+1] and (5N/9(N-1)) (square-root 3N-1.8) < a < (5N/9 (N-1)). (square-root 3N - 0.23), respectively (N is the number of nodes in the network. (3 less-than-or-equal-to N less-than-or-equal-to 10(4)). A class of the distributed routing algorithms for the DOLDL and the implementation procedure of an adaptive fault-tolerant algorithm are proposed. The correctness of the algorithm has been also verified by simulating.

Security Constrained Distributed Optimal Power Flow of Interconnected Power Systems

The security constrained distributed optimal power flow （DOPF） of interconnected power systems is presented. The centralized OPF problem of the multi-area power systems is decomposed into independent DOPF subproblems, one for each area. The dynamic security region （DSR） to guarantee the transient stability constraints and static voltage stability region （SVSR） constraints, and line current limits are included as constraints. The solutions to the DOPF subproblems of the different areas are coordinated through a pricing mechanism until they converge to the centralized OPF solution. The nonlinear DOPF subproblem is solved by predictor-corrector interior point method （PClPM）. The IEEE three-area RTS-96 system is worked out in order to demonstrate the effectiveness of the proposed method.

THE OPTIMAL ACTIVITY DISTRIBUTION IN NONISOTHERMAL PELLETS

The nonisothermal effectiveness fcator for reaction with kinetics r=kc^m/(l+Kc)~a can be improved bycatalysts with nonuniform activity distribution.The optimal distribution function in one-dimensional modelwith which the effectiveness factor can be maximized is a δ-function which means that the activity of thecatalyst should be concentrated on a layer with negligible thickness in a precise locationfrom the centerof pellets.The general equations for predicting the value ofand maximum effectiveness factor as a functionof thermodynamic,kinetic and transport parameters are derived and they can be given explicitly in the case ofa=O,m=a or isothermal reaction.An active layer with definite thickness and a deviation from the optimal locationboth decrease thevalue of the effectiveness factor.It has been shown numerically that the effectiveness factor decreases slightlywith an active layer at the inner side of x but seriously at outer side.

OPTIMAL DYNAMIC LOAD DISTRIBUTION OF MULTIPLE COOPERATING ROBOT MANIPULATORS

For the situation of multiple cooperating manipulators handling a single object,an equilibrium equation is presented in which the manipulator dynamics and control forces/torques are taken into account,and a expression is derived to allow the optimal dynamic load distribution of the combined system can be made.

Distributed Momentum-Based Frank-Wolfe Algorithm for Stochastic Optimization

This paper considers distributed stochastic optimization,in which a number of agents cooperate to optimize a global objective function through local computations and information exchanges with neighbors over a network.Stochastic optimization problems are usually tackled by variants of projected stochastic gradient descent.However,projecting a point onto a feasible set is often expensive.The Frank-Wolfe(FW)method has well-documented merits in handling convex constraints,but existing stochastic FW algorithms are basically developed for centralized settings.In this context,the present work puts forth a distributed stochastic Frank-Wolfe solver,by judiciously combining Nesterov's momentum and gradient tracking techniques for stochastic convex and nonconvex optimization over networks.It is shown that the convergence rate of the proposed algorithm is O(k^(-1/2))for convex optimization,and O(1/log_(2)(k))for nonconvex optimization.The efficacy of the algorithm is demonstrated by numerical simulations against a number of competing alternatives.

An optimization-oriented modeling approach using input convex neural networks and its application on optimal chiller loading

Optimization for the multi-chiller system is an indispensable approach for the operation of highly efficient chiller plants.The optima obtained by model-based optimization algorithms are dependent on precise and solvable objective functions.The classical neural networks cannot provide convex input-output mappings despite capturing impressive nonlinear fitting capabilities,resulting in a reduction in the robustness of model-based optimization.In this paper,we leverage the input convex neural networks(ICNN)to identify the chiller model to construct a convex mapping between control variables and the objective function,which enables the NN-based OCL as a convex optimization problem and apply it to multi-chiller optimization for optimal chiller loading(OCL).Approximation performances are evaluated through a four-model comparison based on an experimental data set,and the statistical results show that,on the premise of retaining prior convexities,the proposed model depicts excellent approximation power for the data set,especially the unseen data.Finally,the ICNN model is applied to a typical OCL problem for a multi-chiller system and combined with three types of optimization strategies.Compared with conventional and meta-heuristic methods,the numerical results suggest that the gradient-based BFGS algorithm provides better energy-saving ratios facing consecutive cooling load inputs and an impressive convergence speed.

Fully asynchronous distributed optimization with linear convergence over directed networks

We study distributed optimization problems over a directed network,where nodes aim to minimize the sum of local objective functions via directed communications with neighbors.Many algorithms are designed to solve it for synchronized or randomly activated implementation,which may create deadlocks in practice.In sharp contrast,we propose a fully asynchronous push-pull gradient(APPG) algorithm,where each node updates without waiting for any other node by using possibly delayed information from neighbors.Then,we construct two novel augmented networks to analyze asynchrony and delays,and quantify its convergence rate from the worst-case point of view.Particularly,all nodes of APPG converge to the same optimal solution at a linear rate of O(λ^(k)) if local functions have Lipschitz-continuous gradients and their sum satisfies the Polyak-?ojasiewicz condition(convexity is not required),where λ ∈(0,1) is explicitly given and the virtual counter k increases by one when any node updates.Finally,the advantage of APPG over the synchronous counterpart and its linear speedup efficiency are numerically validated via a logistic regression problem.

Leader-follower Optimal Selection Method for Distributed Control System in Active Distribution Networks

Aiming at the shortcomings of a traditional centralized control in an active distribution network(AND),this paper proposes a leader-follower distributed group cooperative control strategy to realize multiple operation and control tasks for an ADN.The distributed information exchange protocols of the distributed generation(DG)group devoted to node voltage regulation or exchange power control are developed using a DG power utilization ratio as the consensus variable.On these bases,this study further investigates the leader optimal selection method for a DG group to improve the response speed of the distributed control system.Furthermore,a single or multiple leader selection model is established to minimize the constraints of the one-step convergence factor and the number of leaders to improve the response speed of the distributed control system.The simulation results of the IEEE 33 bus standard test system show the effectiveness of the proposed distributed control strategy.In addition,the response speed of a DG control group can be improved effectively when the single or multiple leaders are selected optimally.

Distributed Robust Optimal Dispatch of Regional Integrated Energy Systems Based on ADMM Algorithm with Adaptive Step Size

This paper proposes a distributed robust optimal dispatch model to enhance information security and interaction among the operators in the regional integrated energy system(RIES).Our model regards the distribution network and each energy hub(EH)as independent operators and employs robust optimization to improve operational security caused by wind and photovoltaic(PV)power output uncertainties,with only deterministic information exchanged across boundaries.This paper also adopts the alternating direction method of multipliers(ADMM)algorithm to facilitate secure information interaction among multiple RIES operators,maximizing the benefit for each subject.Furthermore,the traditional ADMM algorithm with fixed step size is modified to be adaptive,addressing issues of redundant interactions caused by suboptimal initial step size settings.A case study validates the effectiveness of the proposed model,demonstrating the superiority of the ADMM algorithm with adaptive step size and the economic benefits of the distributed robust optimal dispatch model over the distributed stochastic optimal dispatch model.

Optimized Strategy for Layout of Crop Production Areas in Hunan Province

The optimized strategy made a comprehensive consideration of resources, technology, market orientation, production scale, industry basis and layout based on the principle of crop security and farmers’ income increasing, and determined the general planning on layout and structure optimization of future crop production ar-eas, with present crop production, market outlook, future industry development, con-cluding crop production characteristics of the 4 crop regions, and proposing function orientation and highlights.

Cooperative and Competitive Multi-Agent Systems:From Optimization to Games

Multi-agent systems can solve scientific issues related to complex systems that are difficult or impossible for a single agent to solve through mutual collaboration and cooperation optimization.In a multi-agent system,agents with a certain degree of autonomy generate complex interactions due to the correlation and coordination,which is manifested as cooperative/competitive behavior.This survey focuses on multi-agent cooperative optimization and cooperative/non-cooperative games.Starting from cooperative optimization,the studies on distributed optimization and federated optimization are summarized.The survey mainly focuses on distributed online optimization and its application in privacy protection,and overviews federated optimization from the perspective of privacy protection me-chanisms.Then,cooperative games and non-cooperative games are introduced to expand the cooperative optimization problems from two aspects of minimizing global costs and minimizing individual costs,respectively.Multi-agent cooperative and non-cooperative behaviors are modeled by games from both static and dynamic aspects,according to whether each player can make decisions based on the information of other players.Finally,future directions for cooperative optimization,cooperative/non-cooperative games,and their applications are discussed.