Autonomous Vehicle Platoons In Urban Road Networks:A Joint Distributed Reinforcement Learning and Model Predictive Control Approach

In this paper, platoons of autonomous vehicles operating in urban road networks are considered. From a methodological point of view, the problem of interest consists of formally characterizing vehicle state trajectory tubes by means of routing decisions complying with traffic congestion criteria. To this end, a novel distributed control architecture is conceived by taking advantage of two methodologies: deep reinforcement learning and model predictive control. On one hand, the routing decisions are obtained by using a distributed reinforcement learning algorithm that exploits available traffic data at each road junction. On the other hand, a bank of model predictive controllers is in charge of computing the more adequate control action for each involved vehicle. Such tasks are here combined into a single framework:the deep reinforcement learning output(action) is translated into a set-point to be tracked by the model predictive controller;conversely, the current vehicle position, resulting from the application of the control move, is exploited by the deep reinforcement learning unit for improving its reliability. The main novelty of the proposed solution lies in its hybrid nature: on one hand it fully exploits deep reinforcement learning capabilities for decisionmaking purposes;on the other hand, time-varying hard constraints are always satisfied during the dynamical platoon evolution imposed by the computed routing decisions. To efficiently evaluate the performance of the proposed control architecture, a co-design procedure, involving the SUMO and MATLAB platforms, is implemented so that complex operating environments can be used, and the information coming from road maps(links,junctions, obstacles, semaphores, etc.) and vehicle state trajectories can be shared and exchanged. Finally by considering as operating scenario a real entire city block and a platoon of eleven vehicles described by double-integrator models, several simulations have been performed with the aim to put in light the main f eatures of the proposed approach. Moreover, it is important to underline that in different operating scenarios the proposed reinforcement learning scheme is capable of significantly reducing traffic congestion phenomena when compared with well-reputed competitors.

An Optimal Control-Based Distributed Reinforcement Learning Framework for A Class of Non-Convex Objective Functionals of the Multi-Agent Network

This paper studies a novel distributed optimization problem that aims to minimize the sum of the non-convex objective functionals of the multi-agent network under privacy protection, which means that the local objective of each agent is unknown to others. The above problem involves complexity simultaneously in the time and space aspects. Yet existing works about distributed optimization mainly consider privacy protection in the space aspect where the decision variable is a vector with finite dimensions. In contrast, when the time aspect is considered in this paper, the decision variable is a continuous function concerning time. Hence, the minimization of the overall functional belongs to the calculus of variations. Traditional works usually aim to seek the optimal decision function. Due to privacy protection and non-convexity, the Euler-Lagrange equation of the proposed problem is a complicated partial differential equation.Hence, we seek the optimal decision derivative function rather than the decision function. This manner can be regarded as seeking the control input for an optimal control problem, for which we propose a centralized reinforcement learning(RL) framework. In the space aspect, we further present a distributed reinforcement learning framework to deal with the impact of privacy protection. Finally, rigorous theoretical analysis and simulation validate the effectiveness of our framework.

Strengthening mechanisms based on reinforcement distribution uniformity for particle reinforced aluminum matrix composites

A modified mixed strengthening model was proposed for describing the yield strength of particle reinforced aluminum matrix composites.The strengthening mechanisms of the composites were analyzed based on the microstructures and compression mechanical properties.The distribution uniformity of reinforcements and cooperation relationship among dislocation mechanisms were considered in the modified mixed strengthening model by introducing a distribution uniformity factor u and a cooperation coefficient fc,respectively.The results show that the modified mixed strengthening model can accurately describe the yield strengths of Al3Ti/2024Al composites with a relative deviation less than1.2%,which is much more accurate than other strengthening models.The modified mixed model can also be used to predict the yield strength of Al3Ti/2024Al composites with different fractions of reinforcements.

Distributional Reinforcement Learning with Quantum Neural Networks

Traditional reinforcement learning (RL) uses the return, also known as the expected value of cumulative random rewards, for training an agent to learn an optimal policy. However, recent research indicates that learning the distribution over returns has distinct advantages over learning their expected value as seen in different RL tasks. The shift from using the expectation of returns in traditional RL to the distribution over returns in distributional RL has provided new insights into the dynamics of RL. This paper builds on our recent work investigating the quantum approach towards RL. Our work implements the quantile regression (QR) distributional Q learning with a quantum neural network. This quantum network is evaluated in a grid world environment with a different number of quantiles, illustrating its detailed influence on the learning of the algorithm. It is also compared to the standard quantum Q learning in a Markov Decision Process (MDP) chain, which demonstrates that the quantum QR distributional Q learning can explore the environment more efficiently than the standard quantum Q learning. Efficient exploration and balancing of exploitation and exploration are major challenges in RL. Previous work has shown that more informative actions can be taken with a distributional perspective. Our findings suggest another cause for its success: the enhanced performance of distributional RL can be partially attributed to its superior ability to efficiently explore the environment.

Distribution of Reinforcement in Tensile Flanges of Concrete T-shape Continuous Beam

In this paper the analysis of tensile stress distribution in flexural continuous T- beam has been presented. The observed damages in carrying deck of RC bridge over the Wieprz River in Baranow indicate that over pillar zones are not protected enough. The results of numerical analysis have shown that tensile stress in T- section beam appears not only in a web but in flanges as well. Thus reinforcing bars should be distributed within the whole effective width. This fact is mentioned in building codes, for example, in Eurocode 2： ＂Design of concrete structures＂, both in part 1.1 ＂General rules and rules for building＂ and in part 2 ＂Reinforced and prestressed concrete bridges＂, but there are not detailed rules how to place the bars in flanges of T-section.

Mechanical property and deformation mechanism of gold nanowire with non-uniform distribution of twinned boundaries:A molecular dynamics simulation study

The mechanical property and deformation mechanism of twinned gold nanowire with non-uniform distribution of twinned boundaries(TBs)are studied by the molecular dynamics(MD)method.It is found that the twin boundary spacing(TBS)has a great effect on the strength and plasticity of the nanowires with uniform distribution of TBs.And the strength enhances with the decrease of TBS,while its plasticity declines.For the nanowires with non-uniform distribution of TBs,the differences in distribution among different TBSs have little effect on the Young's modulus or strength,and the compromise in strength appears.But the differences have a remarkable effect on the plasticity of twinned gold nanowire.The twinned gold nanowire with higher local symmetry ratio has better plasticity.The initial dislocations always form in the largest TBS and the fracture always appears at or near the twin boundaries adjacent to the smallest TBS.Some simulation results are consistent with the experimental results.

Bond of Seawater Scoria Aggregate Concrete to Stainless Reinforcement

This study investigates the bond between seawater scoria aggregate concrete(SSAC)and stainless reinforcement(SR)through a series of pull-out tests.A total of 39 specimens,considering five experimental parameters—con-crete type(SSAC,ordinary concrete(OC)and seawater coral aggregate concrete(SCAC)),reinforcement type(SR,ordinary reinforcement(OR)),bond length(3,5 and 8 times bar diameter),concrete strength(C25 and C30)and concrete cover thickness(42 and 67 mm)—were prepared.The typical bond properties(failure pattern,bond strength,bond-slip curves and bond stress distribution,etc.)of seawater scoria aggregate concrete-stainless rein-forcement(SSAC-SR)specimen were systematically studied.Generally,the failure pattern changed with the con-crete type used,and the failure surface of SSAC specimen was different from that of OC specimen.SSAC enhanced the bond strength of specimen,while its effect on the deformation of SSAC-SR was negative.On aver-age,the peak slip of SSAC specimens was 20%lower while the bond strength was 6.7%higher compared to OC specimens under the similar conditions.The effects of variables on the bond strength of SSAC–SR in increasing order are concrete type,bond length,concrete strength and cover thickness.The bond-slip curve of SSAC-SR specimen consisted of micro-slipping,slipping and declining stages.It can be obtained that SSAC reduced the curve curvature of bond-slip,and the decline of curve became steep after adopting SR.The typical distribution of bond stress along bond length changed with the types of concrete and reinforcement used.Finally,a specific expression of the bond stress-slip curve considering the effects of various variables was established,which could provide a basis for the practical application of reinforced SSAC.

Double linear strain distribution assumption of RC beam strengthened with external-bonded or near-surface mounted fiber reinforced plastic

Rehabilitation of existing structures with fiber reinforced plastic(FRP)has been growing in popularity because they offer superior performance in terms of resistance to corrosion and high specific stiffness.The strain coordination results of 34 reinforced concrete beams(four groups)strengthened with different methods were presented including external-bonded or near-surface mounted glass or carbon FRP or helical rib bar in order to study the strain coordination of the strengthening materials and steel rebar of RC beam.Because there is relative slipping between concrete and strengthening materials(SM),the strain of SM and steel rebar of RC beam satisfies the double linear strain distribution assumption,that is,the strain of longitudinal fiber parallel to the neutral axis of plated beam within the scope of effective height(h0)of the cross section is in direct proportion to the distance from the fiber to the neutral axis.The strain of SM and steel rebar satisfies the equation εGCH=βεsteel,where the value of β is equal to 1.1-1.3 according to the test results.

Research on Volt/Var Control of Distribution Networks Based on PPO Algorithm

In this paper,a model free volt/var control(VVC)algorithm is developed by using deep reinforcement learning(DRL).We transform the VVC problem of distribution networks into the network framework of PPO algorithm,in order to avoid directly solving a large-scale nonlinear optimization problem.We select photovoltaic inverters as agents to adjust system voltage in a distribution network,taking the reactive power output of inverters as action variables.An appropriate reward function is designed to guide the interaction between photovoltaic inverters and the distribution network environment.OPENDSS is used to output system node voltage and network loss.This method realizes the goal of optimal VVC in distribution network.The IEEE 13-bus three phase unbalanced distribution system is used to verify the effectiveness of the proposed algorithm.Simulation results demonstrate that the proposed method has excellent performance in voltage and reactive power regulation of a distribution network.

RELOCATION ALGORITHM FOR NON-UNIFORM DISTRIBUTION IN MOBILE SENSOR NETWORK

Energy is the determinant factor for the survival of Mobile Sensor Networks(MSN).Based on the analysis of the energy distribution in this paper,a two-phase relocation algorithm is proposed based on the balance between the energy provision and energy consumption distribution.Our main objectives are to maximize the coverage percentage and to minimize the total distance of node movements.This algorithm is designed to meet the requirement of non-uniform distribution network applications,to extend the lifetime of MSN and to simplify the design of the routing protocol.In ad-dition,test results show the feasibility of our proposed relocation algorithm.

Role of matrix structure and flaw size distribution modification on deflection hardening behavior of polyvinyl alcohol fiber reinforced engineered cementitious composites(PVA-ECC)

The multiple cracking and deflection hardening performance of polyvinyl alcohol fiber reinforced engineered cementitious composites(PVA-ECC)under four-point flexural loading have been investigated.Matrices with different binder combinations and W/B ratios(from 0.44 to 0.78)providing satisfactory PVA fiber dispersion were specially designed.Effect of pre-existing flaw size distribution modification on deflection hardening behavior was comparatively studied by adding 3 mm diameter polyethylene beads into the mixtures(6%by total volume).Natural flaw size distributions of composites without beads were determined by cross sectional analysis.The crack number and crack width distributions of specimens after flexural loading were characterized and the possible causes of changes in multiple cracking and deflection hardening behavior by flaw size distribution modification were discussed.Promising results from the view point of deflection hardening behavior were obtained from metakaolin incorporated and flaw size distribution modified PVA-ECCs prepared with W/B=0.53.The dual roles of W/B ratio and superplasticizer content on flaw size distribution,cracking potential and fiber-matrix bond behavior were evaluated.Flaw size distribution modification is found beneficial in terms of ductility improvement at an optimized W/B ratio.

Boundary Data Augmentation for Offline Reinforcement Learning

Offline reinforcement learning(ORL)aims to learn a rational agent purely from behavior data without any online interaction.One of the major challenges encountered in ORL is the problem of distribution shift,i.e.,the mismatch between the knowledge of the learned policy and the reality of the underlying environment.Recent works usually handle this in a too pessimistic manner to avoid out-of-distribution(OOD)queries as much as possible,but this can influence the robustness of the agents at unseen states.In this paper,we propose a simple but effective method to address this issue.The key idea of our method is to enhance the robustness of the new policy learned offline by weakening its confidence in highly uncertain regions,and we propose to find those regions by simulating them with modified Generative Adversarial Nets(GAN)such that the generated data not only follow the same distribution with the old experience but are very difficult to deal with by themselves,with regard to the behavior policy or some other reference policy.We then use this information to regularize the ORL algorithm to penalize the overconfidence behavior in these regions.Extensive experiments on several publicly available offline RL benchmarks demonstrate the feasibility and effectiveness of the proposed method.

Real-time Operation Optimization in Active Distribution Networks Based on Multi-agent Deep Reinforcement Learning

The increasing integration of intermittent renewable energy sources(RESs)poses great challenges to active distribution networks(ADNs),such as frequent voltage fluctuations.This paper proposes a novel ADN strategy based on multiagent deep reinforcement learning(MADRL),which harnesses the regulating function of switch state transitions for the realtime voltage regulation and loss minimization.After deploying the calculated optimal switch topologies,the distribution network operator will dynamically adjust the distributed energy resources(DERs)to enhance the operation performance of ADNs based on the policies trained by the MADRL algorithm.Owing to the model-free characteristics and the generalization of deep reinforcement learning,the proposed strategy can still achieve optimization objectives even when applied to similar but unseen environments.Additionally,integrating parameter sharing(PS)and prioritized experience replay(PER)mechanisms substantially improves the strategic performance and scalability.This framework has been tested on modified IEEE 33-bus,IEEE 118-bus,and three-phase unbalanced 123-bus systems.The results demonstrate the significant real-time regulation capabilities of the proposed strategy.

Simulation of Corrosion-Induced Cracking of Reinforced Concrete Based on Fracture Phase Field Method

Accurate simulation of the cracking process caused by rust expansion of reinforced concrete(RC)structures plays an intuitive role in revealing the corrosion-induced failure mechanism.Considering the quasi-brittle fracture of concrete,the fracture phase field driven by the compressive-shear term is constructed and added to the traditional brittle fracture phase field model.The rationality of the proposed model is verified by a mixed fracture example under a shear displacement load.Then,the extended fracture phase model is applied to simulate the corrosion-induced cracking process of RC.The cracking patterns caused by non-uniform corrosion expansion are discussed for RC specimens with homogeneous macroscopically or heterogeneous with different polygonal aggregate distributions at the mesoscopic scale.Then,the effects of the protective layer on the crack propagation trajectory and cracking resistance are investigated,illustrating that the cracking angle and cracking resistance increase with the increase of the protective layer thickness,consistent with the experimental observation.Finally,the corrosion-induced cracking process of concrete specimens with large and small spacing rebars is simulated,and the interaction of multiple corrosion cracking is easily influenced by the reinforcement spacing,which increases with the decrease of the steel bar interval.These conclusions play an important role in the design of engineering anti-corrosion measures.The fracture phase field model can provide strong support for the life assessment of RC structures.

Decentralized control in active distribution grids via supervised and reinforcement learning

While moving towards a low-carbon, sustainable electricity system, distribution networks are expected to host a large share of distributed generators, such as photovoltaic units and wind turbines. These inverter-based resources are intermittent, but also controllable, and are expected to amplify the role of distribution networks together with other distributed energy resources, such as storage systems and controllable loads. The available control methods for these resources are typically categorized based on the available communication network into centralized, distributed, and decentralized or local. Standard local schemes are typically inefficient, whereas centralized approaches show implementation and cost concerns. This paper focuses on optimized decentralized control of distributed generators via supervised and reinforcement learning. We present existing state-of-the-art decentralized control schemes based on supervised learning, propose a new reinforcement learning scheme based on deep deterministic policy gradient, and compare the behavior of both decentralized and centralized methods in terms of computational effort, scalability, privacy awareness, ability to consider constraints, and overall optimality. We evaluate the performance of the examined schemes on a benchmark European low voltage test system. The results show that both supervised learning and reinforcement learning schemes effectively mitigate the operational issues faced by the distribution network.

Evaluation of Filler Distribution in Particulate Reinforced Composites

Aluminum matrix particulate reinforced composites are of significant interest to industry, but it’s difficult to provide stable properties for this group of material. The mechanical properties of metal matrix composites are deeply influenced by the distribution of reinforcement particulates in the matrix. In this paper uniformity of SiC particles distribution in Al-based composites produced by stir casting and powder metallurgy technique is assessed. Analysis is carried out by means of classical and computer quantification metallographic image analysis methods. In addition, we suggest setting hardness distribution in cross section of samples as an indicator of reinforcement distribution uniformity in the matrix.

Detecting the Resistivity Distribution of Carbon Fiber Reinforced Concrete by Electrical Resistance Tomography Method

According to the principle of electrical resistance tomography （ ERT）, the resistivity distribution of the carbon fiber reinforced concrete （CFRC） in the sensing field can be measured by injecting exciting current and measuring the voltage on the sensor electrode arrays installed on the surface of the object. Therefore, measurement of the resistivity distribution of CFRC is divided into first measuring the boundary conditions and then inversely computing the resistivity distribution. To reach this goal, an ERT system was constructed, which is composed of a sensor array unit, a data acquisition unit and an image reconstruction unit. Simulations of static ERT was performed on set-ups with many objects spread in a homogeneous background, and a simulation of dynamic ERT was also done on a rectangular board, the resistivity of which was changed within a small domain of it. Then, the resistivity distribution of a CFRC sample with a circlar hole as the target was detected by the ERT system. Simulation and experimental results show that the reconstructed ERT image reflects the resistivity distribution or the resistivity change of CFRC structure well. Especially, a small change in resistivity can be identified from the reconstructed images in the simulation of dynamic ERT images.

Study of the Non-Uniform Distribution of Earthquake Risk Within Potential Sources

Potential sources are aggregates of probable future epicenters.In this area,for source models currently,in common use for seismic risk analysis in China,the mean area of each potential source is about 3000-4000 km2.It is assumed that seismic risk has a uniform distribution within the range of each potential source,but studies have shown that the uniform distribution model to a large extent may give an underestimation of the seismic risk.In this paper,the relative distribution of historical epicenters in space within potential sources is discussed,a method is proposed to quantitatively describe the non-uniform distribution of strong earthquakes within potential sources,and some preliminary results are given.By using the results of this paper,seismic risk analysis and seismic zonation can be made more scientific and more reasonable.

Reinforcement Learning with Enhanced Safety for Optimal Dispatch of Distributed Energy Resources in Active Distribution Networks

As numerous distributed energy resources(DERs)are integrated into the distribution networks,the optimal dispatch of DERs is more and more imperative to achieve transition to active distribution networks(ADNs).Since accurate models are usually unavailable in ADNs,an increasing number of reinforcement learning(RL)based methods have been proposed for the optimal dispatch problem.However,these RL based methods are typically formulated without safety guarantees,which hinders their application in real world.In this paper,we propose an RL based method called supervisor-projector-enhanced safe soft actor-critic(S3AC)for the optimal dispatch of DERs in ADNs,which not only minimizes the operational cost but also satisfies safety constraints during online execution.In the proposed S3AC,the data-driven supervisor and projector are pre-trained based on the historical data from supervisory control and data acquisition(SCADA)system,effectively providing enhanced safety for executed actions.Numerical studies on several IEEE test systems demonstrate the effectiveness and safety of the proposed S3AC.

Distribution and engulfment behavior of TiB_2 particles or clusters in wedge-shaped copper casting ingot

Wedge-shaped copper casting experiment was conducted to study the engulfment behavior of TiB2 particle and the interaction between particle or cluster and the solid/liquid front in commercial pure aluminum matrix. The experimental results show that the particle size distribution obeys two separate systems in the whole wedge-cast sample. Furthermore, it is found that the big clusters are pushed to the center of the wedge shaped sample and the single particle or small clusters consisting of few particles are engulfed into the α-Al in the area of the sample edge. The cluster degree of particles varies in different areas, and its value is 0.2 and 0.6 for the cluster fraction in the edge and in the center of the wedge sample, respectively. The cluster diameter does not obey the normal distribution but approximately obeys lognormal distribution in the present work. More importantly, in the whole sample, the particle size obeys two separate log-normal distributions.