Combining reinforcement learning with mathematical programming:An approach for optimal design of heat exchanger networks

Heat integration is important for energy-saving in the process industry.It is linked to the persistently challenging task of optimal design of heat exchanger networks(HEN).Due to the inherent highly nonconvex nonlinear and combinatorial nature of the HEN problem,it is not easy to find solutions of high quality for large-scale problems.The reinforcement learning(RL)method,which learns strategies through ongoing exploration and exploitation,reveals advantages in such area.However,due to the complexity of the HEN design problem,the RL method for HEN should be dedicated and designed.A hybrid strategy combining RL with mathematical programming is proposed to take better advantage of both methods.An insightful state representation of the HEN structure as well as a customized reward function is introduced.A Q-learning algorithm is applied to update the HEN structure using theε-greedy strategy.Better results are obtained from three literature cases of different scales.

Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Coal pillar design when considered a reinforcement problem rather than a suspension problem

Current coal pillar design is the epitome of suspension design.A defined weight of unstable overburden material is estimated, and the dimensions of the pillars left behind are based on holding up that material to a prescribed factor of safety.In principle, this is no different to early roadway roof support design.However, for the most part, roadway roof stabilisation has progressed to reinforcement, whereby the roof strata is assisted in supporting itself.This is now the mainstay of efficient and effective underground coal production.Suspension and reinforcement are fundamentally different in roadway roof stabilisation and lead to substantially different requirements in terms of support hardware characteristics and their application.In suspension, the primary focus is the total load-bearing capacity of the installed support and ensuring that it is securely anchored outside of the unstable roof mass.In contrast, reinforcement recognises that roof de-stabilisation is a gradational process with ever-increasing roof displacement magnitude leading to ever-reducing stability.Key roof support characteristics relate to such issues as system stiffness, the location and pattern of support elements and mobilising a defined thickness of the immediate roof to create(or build) a stabilising strata beam.The objective is to ensure that horizontal stress is maintained at a level that prevents mass roof collapse.This paper presents a prototype coal pillar and overburden system representation where reinforcement, rather than suspension, of the overburden is the stabilising mechanism via the action of in situ horizontal stresses.Established roadway roof reinforcement principles can potentially be applied to coal pillar design under this representation.The merit of this is evaluated according to failed pillar cases as found in a series of published databases.Based on the findings, a series of coal pillar system design considerations for bord and pillar type mine workings are provided.This potentially allows a more flexible approach to coal pillar sizing within workable mining layouts, as compared to common industry practice of a single design factor of safety(Fo S) under defined overburden dead-loading to the exclusion of other relevant overburden stabilising influences.

Energy-Efficient UAV Trajectory Design for Backscatter Communication: A Deep Reinforcement Learning Approach

Recently,backscatter communication(BC)has been introduced as a green paradigm for Internet of Things(IoT).Meanwhile,unmanned aerial vehicles(UAVs)can serve as aerial base stations(BSs)to enhance the performance of BC system thanks to their high mobility and flexibility.In this paper,we investigate the problem of energy efficiency(EE)for an energy-limited backscatter communication(BC)network,where backscatter devices(BDs)on the ground harvest energy from the wireless signal of a flying rotary-wing quadrotor.Specifically,we first reformulate the EE optimization problem as a Markov decision process(MDP)and then propose a deep reinforcement learning(DRL)algorithm to design the UAV trajectory with the constraints of the BD scheduling,the power reflection coefficients,the transmission power,and the fairness among BDs.Simulation results show the proposed DRL algorithm achieves close-to-optimal performance and significant EE gains compared to the benchmark schemes.

Shape design of the reinforcement for bending load-carrying capacity of under-matched butt joint under four-point bending load

To improve the bending load-carrying capacity （ BLCC） of under-matched butt joint under four-point bending load in the elastic stage, the shape design of the reinforcement is studied based on the theoretics of mechanics of materials. The concept, criterion, realization condition and design proposal of equal bending load-carrying capacity （EBLCC） are put forward. The theoretical analysis results have been verified by the finite element method. The simulation results are coincident basically with the ones of theoretical analysis. The research results show that the shape design of the reinforcement of EBLCC can improve BLCC of under-matched butt joint and the unilateral-side type reinforcement can replace double-side symmetry

Multi-Agent Few-Shot Meta Reinforcement Learning for Trajectory Design and Channel Selection in UAV-Assisted Networks

Unmanned aerial vehicle(UAV)-assisted communications have been considered as a solution of aerial networking in future wireless networks due to its low-cost, high-mobility, and swift features. This paper considers a UAV-assisted downlink transmission,where UAVs are deployed as aerial base stations to serve ground users. To maximize the average transmission rate among the ground users, this paper formulates a joint optimization problem of UAV trajectory design and channel selection, which is NP-hard and non-convex. To solve the problem, we propose a multi-agent deep Q-network(MADQN) scheme.Specifically, the agents that the UAVs act as perform actions from their observations distributively and share the same reward. To tackle the tasks where the experience is insufficient, we propose a multi-agent meta reinforcement learning algorithm to fast adapt to the new tasks. By pretraining the tasks with similar distribution, the learning model can acquire general knowledge. Simulation results have indicated the MADQN scheme can achieve higher throughput than fixed allocation. Furthermore, our proposed multiagent meta reinforcement learning algorithm learns the new tasks much faster compared with the MADQN scheme.

Deep Reinforcement Learning for Multi-Phase Microstructure Design

This paper presents a de-novo computational design method driven by deep reinforcement learning to achieve reliable predictions and optimum properties for periodic microstructures.With recent developments in 3-D printing,microstructures can have complex geometries and material phases fabricated to achieve targeted mechanical performance.These material property enhancements are promising in improving the mechanical,thermal,and dynamic performance in multiple engineering systems,ranging from energy harvesting applications to spacecraft components.The study investigates a novel and efficient computational framework that integrates deep reinforcement learning algorithms into finite element-based material simulations to quantitatively model and design 3-D printed periodic microstructures.These algorithms focus on improving the mechanical and thermal performance of engineering components by optimizing a microstructural architecture to meet different design requirements.Additionally,the machine learning solutions demonstrated equivalent results to the physics-based simulations while significantly improving the computational time efficiency.The outcomes of the project show promise to the automation of the design and manufacturing of microstructures to enable their fabrication in large quantities with the utilization of the 3-D printing technology.

Design and Construction Technology of Prefabricated Reinforced Concrete Slab Culverts

Compared with traditional cast-in-situ concrete slab culverts,prefabricated reinforced concrete slab culverts can be produced more quickly and has strong quality controllability,strong earthquake resistance,and repeatability.They will be the primary production method of slab culverts in the future.This article offers a comprehensive review of the design and construction technology associated with prefabricated reinforced concrete slab culverts.The objective is to provide a valuable reference for related enterprises,enhance the quality of design and construction in precast pile configuration,and,in turn,contribute to the advancement of construction projects within our country.

Multi-Scale Design and Optimization of Composite Material Structure for Heavy-Duty Truck Protection Device

In this paper,to present a lightweight-developed front underrun protection device(FUPD)for heavy-duty trucks,plain weave carbon fiber reinforced plastic(CFRP)is used instead of the original high-strength steel.First,the mechanical and structural properties of plain carbon fiber composite anti-collision beams are comparatively analyzed from a multi-scale perspective.For studying the design capability of carbon fiber composite materials,we investigate the effects of TC-33 carbon fiber diameter(D),fiber yarn width(W)and height(H),and fiber yarn density(N)on the front underrun protective beam of carbon fiber compositematerials.Based on the investigation,a material-structure matching strategy suitable for the front underrun protective beam of heavy-duty trucks is proposed.Next,the composite material structure is optimized by applying size optimization and stack sequence optimization methods to obtain the higher performance carbon fiber composite front underrun protection beam of commercial vehicles.The results show that the fiber yarn height(H)has the greatest influence on the protective beam,and theH1matching scheme for the front underrun protective beamwith a carbon fiber composite structure exhibits superior performance.The proposed method achieves a weight reduction of 55.21% while still meeting regulatory requirements,which demonstrates its remarkable weight reduction effect.

Study on determining of reinforced concrete false roof strength and design of reinforcement based on reliability theory

Study on efficient mining of the steep incline and fractured ore-bodies in Yongshaba mine of Guizhou Kailin Group shows that ore-body is fractured and difficult to support the roadways in-vein.After research of the actual conditions about the ore-bodies,we have made the initial decision to adopt reconstruction of roof downward sublevel cut-and-fill mining.The men work safely under the false roof supporting the top plate.However,the difficult problem is how to determine the strength of the false roof.In this case,the method based on reliability theory has been put forward.Combined with elastic mechanics and field practice,when practical value of reliable probability is 90 %,the value of the false roof strength has been calculated,and the study shows that stope span greatly influences the false roof strength.With the strength of artificial roof,the reasonable reinforcement design ensures the false roof which can supply the demand of strength under large span and load.

Trajectory Design for UAV-Enabled Maritime Secure Communications:A Reinforcement Learning Approach

This paper investigates an unmanned aerial vehicle(UAV)-enabled maritime secure communication network,where the UAV aims to provide the communication service to a legitimate mobile vessel in the presence of multiple eavesdroppers.In this maritime communication networks(MCNs),it is challenging for the UAV to determine its trajectory on the ocean,since it cannot land or replenish energy on the sea surface,the trajectory should be pre-designed before the UAV takes off.Furthermore,the take-off location of the UAV and the sea lane of the vessel may be random,which leads to a highly dynamic environment.To address these issues,we propose two reinforcement learning schemes,Q-learning and deep deterministic policy gradient(DDPG)algorithms,to solve the discrete and continuous UAV trajectory design problem,respectively.Simulation results are provided to validate the effectiveness and superior performance of the proposed reinforcement learning schemes versus the existing schemes in the literature.Additionally,the proposed DDPG algorithm converges faster and achieves higher utilities for the UAV,compared to the Q-learning algorithm.

Studies on Seismic Identification and Reinforcement Design of Building Structures

China is a country with many earthquakes. Seismic safety monitoring and building earthquake-proofing technique are important means to protect the safety of people’s property in China. However, up to now, China’s seismic reinforcement and identification technology is still not mature enough. In particular, the 2008 Wenchuan earthquake caused great loss of life and safety to the Chinese people. This paper, takes seismic identification and reinforcement technology of building structures as the research object and summarizes the main methods of building structure seismic resistance in China. This paper is based on an in-depth analysis of the main seismic reinforcement and identification techniques in China, deeply analyzes the crux of anti-seismic and reinforcement of building structure combining with the current building seismic reinforcement typical cases, and puts forward some reasonable suggestions and improvement methods for the future development of building seismic identification and reinforcement design.

Reinforcement of Soft Foundation with Geotextile and Observation for Sea Dike Project of Zhapu Port

The design method of reinforcement of soft foundation with geotextile for the sea dike of the Zhapu Port is discussed in this paper. The prototype behaviours such as pore water pressure, settlement and so on were observed. The degree of consolidation is found out from observed pore water pressure and observed settlement respectively, then the strength increment of soil is calculated and compared with that obtained from vane shear tests. For the use of observed pore water pressure, the consolidation coefficient of soil is deduced approximately with a method named experimental exponential interpolation. The degree of consolidation of the ground is deduced theoretically from the dissipation of pore water pressure. Besides, the logarithmic curve and hyperbola are used to fit the observed time-settlement curve, and the degree of consolidation of soil is obtained according to the definition of the consolidation degree. After preliminary verification with observed prototype data, the method to reinforce the low dike with geotextile is considered to be simple and rational, and it can also reduce the construction cost.

Exploring Local Chemical Space in De Novo Molecular Generation Using Multi-Agent Deep Reinforcement Learning

Single-agent reinforcement learning (RL) is commonly used to learn how to play computer games, in which the agent makes one move before making the next in a sequential decision process. Recently single agent was also employed in the design of molecules and drugs. While a single agent is a good fit for computer games, it has limitations when used in molecule design. Its sequential learning makes it impossible to modify or improve the previous steps while working on the current step. In this paper, we proposed to apply the multi-agent RL approach to the research of molecules, which can optimize all sites of a molecule simultaneously. To elucidate the validity of our approach, we chose one chemical compound Favipiravir to explore its local chemical space. Favipiravir is a broad-spectrum inhibitor of viral RNA polymerase, and is one of the compounds that are currently being used in SARS-CoV-2 (COVID-19) clinical trials. Our experiments revealed the collaborative learning of a team of deep RL agents as well as the learning of its individual learning agent in the exploration of Favipiravir. In particular, our multi-agents not only discovered the molecules near Favipiravir in chemical space, but also the learnability of each site in the string representation of Favipiravir, critical information for us to understand the underline mechanism that supports machine learning of molecules.

Improving the design of reinforcing frames by simulating the arch and peltate venation structures

Based on the analyses on arch and peltate venation structures, the design of reinforcing frames was improved. First, distribution rules of the arch structure were summarized. According to the load condition and the structure of the frame, a mechanical model of arch structure was devel- oped, and two solutions for the model were analyzed and compared with each other. Through the a- nalysis, application rules of arch structure for improving the design were obtained. Then, distribu- tion rules of peltate venation structure were summarized. By using the same method, application rules of peltate venation structure for improving the design were also obtained. Finally, mechanical problem of the frame was described, and rib arrangement of the frame was redesigned. A parameter optimization for the widths of ribs in bionic arrangement was also carried out to accomplish the im- proving design. Comparison between bionic and conventional reinforcing frames shows that the weight is reduced by as much as 15.3%.

Optimal seismic design of reinforced concrete structures under timehistory earthquake loads using an intelligent hybrid algorithm

A reliable seismic-resistant design of structures is achieved in accordance with the seismic design codes by designing structures under seven or more pairs of earthquake records. Based on the recommendations of seismic design codes, the average time-history responses （ATHR） of structure is required. This paper focuses on the optimal seismic design of reinforced concrete （RC） structures against ten earthquake records using a hybrid of particle swarm optimization algorithm and an intelligent regression model （IRM）. In order to reduce the computational time of optimization procedure due to the computational efforts of time-history analyses, IRM is proposed to accurately predict ATHR of structures. The proposed IRM consists of the combination of the subtractive algorithm （SA）, K-means clustering approach and wavelet weighted least squares support vector machine （WWLS-SVM）. To predict ATHR of structures, first, the input-output samples of structures are classified by SA and K-means clustering approach. Then, WWLS-SVM is trained with few samples and high accuracy for each cluster. 9- and 18-storey RC frames are designed optimally to illustrate the effectiveness and practicality of the proposed IRM. The numerical results demonstrate the efficiency and computational advantages of IRM for optimal design of structures subjected to time-history earthquake loads.

Finite Element Method for Design of Reinforced Concrete Offshore Platforms

A design method of reinforced concrete (R. C.) offshore platforms with nonlinear finite element analysis is proposed. According to the method, a computer program is developed. In this program nonlinear constitutive relationships and strength criteria of concrete and steel bars are included, and the progressive cracking and crushing of the concrete are taken into account. Based on the stress distribution obtained by the nonlinear finite element analysis, the amount of reinforcement in the control sections can be computed and adjusted automatically by the program to satisfy the requirement of the design. The amount of reinforcement required in the control sections, which are obtained with the nonlinear finite element analysis, is agreeable to that obtained in the experiment. This shows that the design method of R. C. offshore platform with the nonlinear finite element method proposed by the authors is reliable for practical use.

Advanced Welding Technology for Highly Stressable Multi-Material Designs with Fiber-Reinforced Plastics and Metals

Organic sheets made out of fiber-reinforced thermoplastics are able to make a crucial contribution to increase the lightweight potential of a design. They show high specific strength- and stiffness properties, good damping characteristics and recycling capabilities, while being able to show a higher energy absorption capacity than comparable metal constructions. Nowadays, multi-material designs are an established way in the automotive industry to combine the benefits of metal and fiber-reinforced plastics. Currently used technologies for the joining of organic sheets and metals in large-scale production are mechanical joining technologies and adhesive technologies. Both techniques require large overlapping areas that are not required in the design of the part. Additionally, mechanical joining is usually combined with “fiber-destroying” pre-drilling and punching processes. This will disturb the force flux at the joining location by causing unwanted fiber- and inter-fiber failure and inducing critical notch stresses. Therefore, the multi-material design with fiber-reinforced thermoplastics and metals needs optimized joining techniques that don’t interrupt the force flux, so that higher loads can be induced and the full benefit of the FRP material can be used. This article focuses on the characterization of a new joining technology, based on the Cold Metal Transfer (CMT) welding process that allows joining of organic sheets and metals in a load path optimized way, with short cycle times. This is achieved by redirecting the fibers around the joining area by the insertion of a thin metal pin. The path of the fibers will be similar to paths of fibers inside structures found in nature, e.g. a knothole inside of a tree. As a result of the bionic fiber design of the joint, high joining strengths can be achieved. The increase of the joint strength compared to blind riveting was performed and proven with stainless steel and orthotropic reinforced composites in shear-tests based on the DIN EN ISO 14273. Every specimen joined with the new CMT Pin joining technology showed a higher strength than specimens joined with one blind rivet. Specimens joined with two or three pin rows show a higher strength than specimens joined with two blind rivets.

Structural Topology Optimization by Combining BESO with Reinforcement Learning

In this paper,a new algorithm combining the features of bi-direction evolutionary structural optimization(BESO)and reinforcement learning(RL)is proposed for continuum structural topology optimization(STO).In contrast to conventional approaches which only generate a certain quasi-optimal solution,the goal of the combined method is to provide more quasi-optimal solutions for designers such as the idea of generative design.Two key components were adopted.First,besides sensitivity,value function updated by Monte-Carlo reinforcement learning was utilized to measure the importance of each element,which made the solving process convergent and closer to the optimum.Second,ε-greedy policy added a random perturbation to the main search direction so as to extend the search ability.Finally,the quality and diversity of solutions could be guaranteed by controlling the value of compliance as well as Intersection-over-Union(IoU).Results of several 2D and 3D compliance minimization problems,including a geometrically nonlinear case,show that the combined method is capable of generating a group of good and different solutions that satisfy various possible requirements in engineering design within acceptable computation cost.

Reinforcement Learning of Molecule Optimization with Bayesian Neural Networks

Creating new molecules with desired properties is a fundamental and challenging problem in chemistry. Reinforcement learning (RL) has shown its utility in this area where the target chemical property values can serve as a reward signal. At each step of making a new molecule, the RL agent learns selecting an action from a list of many chemically valid actions for a given molecule, implying a great uncertainty associated with its learning. In a traditional implementation of deep RL algorithms, deterministic neural networks are typically employed, thus allowing the agent to choose one action from one sampled action at each step. In this paper, we proposed a new strategy of applying Bayesian neural networks to RL to reduce uncertainty so that the agent can choose one action from a pool of sampled actions at each step, and investigated its benefits in molecule design. Our experiments suggested the Bayesian approach could create molecules of desirable chemical quality while maintained their diversity, a very difficult goal to achieve in machine learning of molecules. We further exploited their diversity by using them to train a generative model to yield more novel drug-like molecules, which were absent in the training molecules as we know novelty is essential for drug candidate molecules. In conclusion, Bayesian approach could offer a balance between exploitation and exploration in RL, and a balance between optimization and diversity in molecule design.