Combining reinforcement learning with mathematical programming:An approach for optimal design of heat exchanger networks

Heat integration is important for energy-saving in the process industry.It is linked to the persistently challenging task of optimal design of heat exchanger networks(HEN).Due to the inherent highly nonconvex nonlinear and combinatorial nature of the HEN problem,it is not easy to find solutions of high quality for large-scale problems.The reinforcement learning(RL)method,which learns strategies through ongoing exploration and exploitation,reveals advantages in such area.However,due to the complexity of the HEN design problem,the RL method for HEN should be dedicated and designed.A hybrid strategy combining RL with mathematical programming is proposed to take better advantage of both methods.An insightful state representation of the HEN structure as well as a customized reward function is introduced.A Q-learning algorithm is applied to update the HEN structure using theε-greedy strategy.Better results are obtained from three literature cases of different scales.

Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

The Ultimate Anti-Seismic Design Method

The design mechanisms and methods of the invention are intended to minimize problems related to the safety of structures in the event of natural phenomena such as earthquakes, tornadoes, and strong winds. It is achieved by controlling the deformations of the structure. Damage and deformation are closely related concepts since the control of deformations also controls the damage. The design method of applying artificial compression to the ends of all longitudinal reinforced concrete walls and, at the same time, connecting the ends of the walls to the ground using ground anchors placed at the depths of the boreholes, transfers the inertial stresses of the structure in the ground, which reacts as an external force in the structure’s response to seismic displacements. The wall with the artificial compression acquires dynamic, larger active cross-section and high axial and torsional stiffness, preventing all failures caused by inelastic deformation. By connecting the ends of all walls to the ground, we control the eigenfrequency of the structure and the ground during each seismic loading cycle, preventing inelastic displacements. At the same time, we ensure the strong bearing capacity of the foundation soil and the structure. By designing the walls correctly and placing them in proper locations, we prevent the torsional flexural buckling that occurs in asymmetrical floor plans, and metal and tall structures. Compression of the wall sections at the ends and their anchoring to the ground mitigates the transfer of deformations to the connection nodes, strengthens the wall section in terms of base shear force and shear stress of the sections, and increases the strength of the cross-sections to the tensile at the ends of the walls by introducing counteractive forces. The use of tendons within the ducts prevents longitudinal shear in the overlay concrete, while anchoring the walls to the foundation not only dissipates inertial forces to the ground but also prevents rotation of the walls, thus maintaining the structural integrity of the beams. The prestressing at the bilateral ends of the walls restores the structure to its original position even inelastic displacements by closing the opening of the developing cracks.

Construction of Teaching Case Base of Anti-Seismic Design of Building Structure

Anti-Seismic Design of Building Structures is an important course in civil engineering majors,and it is also a course that pays equal attention to theory and practice.Therefore,by establishing a case base for Anti-Seismic Design of Building Structures,the obscure theoretical knowledge can be taught to students in the form of examples,and the knowledge becomes intuitive.In this way,the students’understanding of anti-seismic design theory and the efficiency of teaching can be improved,and the students’interest in learning can be stimulated.

“3 Degrees and 8 Combinations” Teaching Mode of Anti-Seismic Design of Building Structures

Earthquakes pose a significant threat to people’s property and personal safety.Improving the teaching of civil engineering and building structure anti-seismic design courses can enable students to do a good job in anti-seismic design in the future and effectively reduce the damage on buildings caused by earthquakes.In this paper,we analyzed the basic characteristics of a course in civil engineering major,which is Anti-Seismic Design of Building Structures,and the shortcomings of traditional teaching.It is proposed that the 3-degrees and 8-combinations teaching mode of anti-seismic design of building structures can effectively improve students’autonomy and enthusiasm in learning,helps to cultivate professional ethics among students,and improve their ability to apply what they have learned.

Coal pillar design when considered a reinforcement problem rather than a suspension problem

Current coal pillar design is the epitome of suspension design.A defined weight of unstable overburden material is estimated, and the dimensions of the pillars left behind are based on holding up that material to a prescribed factor of safety.In principle, this is no different to early roadway roof support design.However, for the most part, roadway roof stabilisation has progressed to reinforcement, whereby the roof strata is assisted in supporting itself.This is now the mainstay of efficient and effective underground coal production.Suspension and reinforcement are fundamentally different in roadway roof stabilisation and lead to substantially different requirements in terms of support hardware characteristics and their application.In suspension, the primary focus is the total load-bearing capacity of the installed support and ensuring that it is securely anchored outside of the unstable roof mass.In contrast, reinforcement recognises that roof de-stabilisation is a gradational process with ever-increasing roof displacement magnitude leading to ever-reducing stability.Key roof support characteristics relate to such issues as system stiffness, the location and pattern of support elements and mobilising a defined thickness of the immediate roof to create(or build) a stabilising strata beam.The objective is to ensure that horizontal stress is maintained at a level that prevents mass roof collapse.This paper presents a prototype coal pillar and overburden system representation where reinforcement, rather than suspension, of the overburden is the stabilising mechanism via the action of in situ horizontal stresses.Established roadway roof reinforcement principles can potentially be applied to coal pillar design under this representation.The merit of this is evaluated according to failed pillar cases as found in a series of published databases.Based on the findings, a series of coal pillar system design considerations for bord and pillar type mine workings are provided.This potentially allows a more flexible approach to coal pillar sizing within workable mining layouts, as compared to common industry practice of a single design factor of safety(Fo S) under defined overburden dead-loading to the exclusion of other relevant overburden stabilising influences.

Energy-Efficient UAV Trajectory Design for Backscatter Communication: A Deep Reinforcement Learning Approach

Recently,backscatter communication(BC)has been introduced as a green paradigm for Internet of Things(IoT).Meanwhile,unmanned aerial vehicles(UAVs)can serve as aerial base stations(BSs)to enhance the performance of BC system thanks to their high mobility and flexibility.In this paper,we investigate the problem of energy efficiency(EE)for an energy-limited backscatter communication(BC)network,where backscatter devices(BDs)on the ground harvest energy from the wireless signal of a flying rotary-wing quadrotor.Specifically,we first reformulate the EE optimization problem as a Markov decision process(MDP)and then propose a deep reinforcement learning(DRL)algorithm to design the UAV trajectory with the constraints of the BD scheduling,the power reflection coefficients,the transmission power,and the fairness among BDs.Simulation results show the proposed DRL algorithm achieves close-to-optimal performance and significant EE gains compared to the benchmark schemes.

Shape design of the reinforcement for bending load-carrying capacity of under-matched butt joint under four-point bending load

To improve the bending load-carrying capacity （ BLCC） of under-matched butt joint under four-point bending load in the elastic stage, the shape design of the reinforcement is studied based on the theoretics of mechanics of materials. The concept, criterion, realization condition and design proposal of equal bending load-carrying capacity （EBLCC） are put forward. The theoretical analysis results have been verified by the finite element method. The simulation results are coincident basically with the ones of theoretical analysis. The research results show that the shape design of the reinforcement of EBLCC can improve BLCC of under-matched butt joint and the unilateral-side type reinforcement can replace double-side symmetry

Multi-Agent Few-Shot Meta Reinforcement Learning for Trajectory Design and Channel Selection in UAV-Assisted Networks

Unmanned aerial vehicle(UAV)-assisted communications have been considered as a solution of aerial networking in future wireless networks due to its low-cost, high-mobility, and swift features. This paper considers a UAV-assisted downlink transmission,where UAVs are deployed as aerial base stations to serve ground users. To maximize the average transmission rate among the ground users, this paper formulates a joint optimization problem of UAV trajectory design and channel selection, which is NP-hard and non-convex. To solve the problem, we propose a multi-agent deep Q-network(MADQN) scheme.Specifically, the agents that the UAVs act as perform actions from their observations distributively and share the same reward. To tackle the tasks where the experience is insufficient, we propose a multi-agent meta reinforcement learning algorithm to fast adapt to the new tasks. By pretraining the tasks with similar distribution, the learning model can acquire general knowledge. Simulation results have indicated the MADQN scheme can achieve higher throughput than fixed allocation. Furthermore, our proposed multiagent meta reinforcement learning algorithm learns the new tasks much faster compared with the MADQN scheme.

Deep Reinforcement Learning for Multi-Phase Microstructure Design

This paper presents a de-novo computational design method driven by deep reinforcement learning to achieve reliable predictions and optimum properties for periodic microstructures.With recent developments in 3-D printing,microstructures can have complex geometries and material phases fabricated to achieve targeted mechanical performance.These material property enhancements are promising in improving the mechanical,thermal,and dynamic performance in multiple engineering systems,ranging from energy harvesting applications to spacecraft components.The study investigates a novel and efficient computational framework that integrates deep reinforcement learning algorithms into finite element-based material simulations to quantitatively model and design 3-D printed periodic microstructures.These algorithms focus on improving the mechanical and thermal performance of engineering components by optimizing a microstructural architecture to meet different design requirements.Additionally,the machine learning solutions demonstrated equivalent results to the physics-based simulations while significantly improving the computational time efficiency.The outcomes of the project show promise to the automation of the design and manufacturing of microstructures to enable their fabrication in large quantities with the utilization of the 3-D printing technology.

Design and Construction Technology of Prefabricated Reinforced Concrete Slab Culverts

Compared with traditional cast-in-situ concrete slab culverts,prefabricated reinforced concrete slab culverts can be produced more quickly and has strong quality controllability,strong earthquake resistance,and repeatability.They will be the primary production method of slab culverts in the future.This article offers a comprehensive review of the design and construction technology associated with prefabricated reinforced concrete slab culverts.The objective is to provide a valuable reference for related enterprises,enhance the quality of design and construction in precast pile configuration,and,in turn,contribute to the advancement of construction projects within our country.

Multi-Scale Design and Optimization of Composite Material Structure for Heavy-Duty Truck Protection Device

In this paper,to present a lightweight-developed front underrun protection device(FUPD)for heavy-duty trucks,plain weave carbon fiber reinforced plastic(CFRP)is used instead of the original high-strength steel.First,the mechanical and structural properties of plain carbon fiber composite anti-collision beams are comparatively analyzed from a multi-scale perspective.For studying the design capability of carbon fiber composite materials,we investigate the effects of TC-33 carbon fiber diameter(D),fiber yarn width(W)and height(H),and fiber yarn density(N)on the front underrun protective beam of carbon fiber compositematerials.Based on the investigation,a material-structure matching strategy suitable for the front underrun protective beam of heavy-duty trucks is proposed.Next,the composite material structure is optimized by applying size optimization and stack sequence optimization methods to obtain the higher performance carbon fiber composite front underrun protection beam of commercial vehicles.The results show that the fiber yarn height(H)has the greatest influence on the protective beam,and theH1matching scheme for the front underrun protective beamwith a carbon fiber composite structure exhibits superior performance.The proposed method achieves a weight reduction of 55.21% while still meeting regulatory requirements,which demonstrates its remarkable weight reduction effect.

Study on determining of reinforced concrete false roof strength and design of reinforcement based on reliability theory

Study on efficient mining of the steep incline and fractured ore-bodies in Yongshaba mine of Guizhou Kailin Group shows that ore-body is fractured and difficult to support the roadways in-vein.After research of the actual conditions about the ore-bodies,we have made the initial decision to adopt reconstruction of roof downward sublevel cut-and-fill mining.The men work safely under the false roof supporting the top plate.However,the difficult problem is how to determine the strength of the false roof.In this case,the method based on reliability theory has been put forward.Combined with elastic mechanics and field practice,when practical value of reliable probability is 90 %,the value of the false roof strength has been calculated,and the study shows that stope span greatly influences the false roof strength.With the strength of artificial roof,the reasonable reinforcement design ensures the false roof which can supply the demand of strength under large span and load.

Trajectory Design for UAV-Enabled Maritime Secure Communications:A Reinforcement Learning Approach

This paper investigates an unmanned aerial vehicle(UAV)-enabled maritime secure communication network,where the UAV aims to provide the communication service to a legitimate mobile vessel in the presence of multiple eavesdroppers.In this maritime communication networks(MCNs),it is challenging for the UAV to determine its trajectory on the ocean,since it cannot land or replenish energy on the sea surface,the trajectory should be pre-designed before the UAV takes off.Furthermore,the take-off location of the UAV and the sea lane of the vessel may be random,which leads to a highly dynamic environment.To address these issues,we propose two reinforcement learning schemes,Q-learning and deep deterministic policy gradient(DDPG)algorithms,to solve the discrete and continuous UAV trajectory design problem,respectively.Simulation results are provided to validate the effectiveness and superior performance of the proposed reinforcement learning schemes versus the existing schemes in the literature.Additionally,the proposed DDPG algorithm converges faster and achieves higher utilities for the UAV,compared to the Q-learning algorithm.

Studies on Seismic Identification and Reinforcement Design of Building Structures

China is a country with many earthquakes. Seismic safety monitoring and building earthquake-proofing technique are important means to protect the safety of people’s property in China. However, up to now, China’s seismic reinforcement and identification technology is still not mature enough. In particular, the 2008 Wenchuan earthquake caused great loss of life and safety to the Chinese people. This paper, takes seismic identification and reinforcement technology of building structures as the research object and summarizes the main methods of building structure seismic resistance in China. This paper is based on an in-depth analysis of the main seismic reinforcement and identification techniques in China, deeply analyzes the crux of anti-seismic and reinforcement of building structure combining with the current building seismic reinforcement typical cases, and puts forward some reasonable suggestions and improvement methods for the future development of building seismic identification and reinforcement design.

Reinforcement of Soft Foundation with Geotextile and Observation for Sea Dike Project of Zhapu Port

The design method of reinforcement of soft foundation with geotextile for the sea dike of the Zhapu Port is discussed in this paper. The prototype behaviours such as pore water pressure, settlement and so on were observed. The degree of consolidation is found out from observed pore water pressure and observed settlement respectively, then the strength increment of soil is calculated and compared with that obtained from vane shear tests. For the use of observed pore water pressure, the consolidation coefficient of soil is deduced approximately with a method named experimental exponential interpolation. The degree of consolidation of the ground is deduced theoretically from the dissipation of pore water pressure. Besides, the logarithmic curve and hyperbola are used to fit the observed time-settlement curve, and the degree of consolidation of soil is obtained according to the definition of the consolidation degree. After preliminary verification with observed prototype data, the method to reinforce the low dike with geotextile is considered to be simple and rational, and it can also reduce the construction cost.

Exploring Local Chemical Space in De Novo Molecular Generation Using Multi-Agent Deep Reinforcement Learning

Single-agent reinforcement learning (RL) is commonly used to learn how to play computer games, in which the agent makes one move before making the next in a sequential decision process. Recently single agent was also employed in the design of molecules and drugs. While a single agent is a good fit for computer games, it has limitations when used in molecule design. Its sequential learning makes it impossible to modify or improve the previous steps while working on the current step. In this paper, we proposed to apply the multi-agent RL approach to the research of molecules, which can optimize all sites of a molecule simultaneously. To elucidate the validity of our approach, we chose one chemical compound Favipiravir to explore its local chemical space. Favipiravir is a broad-spectrum inhibitor of viral RNA polymerase, and is one of the compounds that are currently being used in SARS-CoV-2 (COVID-19) clinical trials. Our experiments revealed the collaborative learning of a team of deep RL agents as well as the learning of its individual learning agent in the exploration of Favipiravir. In particular, our multi-agents not only discovered the molecules near Favipiravir in chemical space, but also the learnability of each site in the string representation of Favipiravir, critical information for us to understand the underline mechanism that supports machine learning of molecules.

Improving the design of reinforcing frames by simulating the arch and peltate venation structures

Based on the analyses on arch and peltate venation structures, the design of reinforcing frames was improved. First, distribution rules of the arch structure were summarized. According to the load condition and the structure of the frame, a mechanical model of arch structure was devel- oped, and two solutions for the model were analyzed and compared with each other. Through the a- nalysis, application rules of arch structure for improving the design were obtained. Then, distribu- tion rules of peltate venation structure were summarized. By using the same method, application rules of peltate venation structure for improving the design were also obtained. Finally, mechanical problem of the frame was described, and rib arrangement of the frame was redesigned. A parameter optimization for the widths of ribs in bionic arrangement was also carried out to accomplish the im- proving design. Comparison between bionic and conventional reinforcing frames shows that the weight is reduced by as much as 15.3%.

Optimal seismic design of reinforced concrete structures under timehistory earthquake loads using an intelligent hybrid algorithm

A reliable seismic-resistant design of structures is achieved in accordance with the seismic design codes by designing structures under seven or more pairs of earthquake records. Based on the recommendations of seismic design codes, the average time-history responses （ATHR） of structure is required. This paper focuses on the optimal seismic design of reinforced concrete （RC） structures against ten earthquake records using a hybrid of particle swarm optimization algorithm and an intelligent regression model （IRM）. In order to reduce the computational time of optimization procedure due to the computational efforts of time-history analyses, IRM is proposed to accurately predict ATHR of structures. The proposed IRM consists of the combination of the subtractive algorithm （SA）, K-means clustering approach and wavelet weighted least squares support vector machine （WWLS-SVM）. To predict ATHR of structures, first, the input-output samples of structures are classified by SA and K-means clustering approach. Then, WWLS-SVM is trained with few samples and high accuracy for each cluster. 9- and 18-storey RC frames are designed optimally to illustrate the effectiveness and practicality of the proposed IRM. The numerical results demonstrate the efficiency and computational advantages of IRM for optimal design of structures subjected to time-history earthquake loads.

Finite Element Method for Design of Reinforced Concrete Offshore Platforms

A design method of reinforced concrete (R. C.) offshore platforms with nonlinear finite element analysis is proposed. According to the method, a computer program is developed. In this program nonlinear constitutive relationships and strength criteria of concrete and steel bars are included, and the progressive cracking and crushing of the concrete are taken into account. Based on the stress distribution obtained by the nonlinear finite element analysis, the amount of reinforcement in the control sections can be computed and adjusted automatically by the program to satisfy the requirement of the design. The amount of reinforcement required in the control sections, which are obtained with the nonlinear finite element analysis, is agreeable to that obtained in the experiment. This shows that the design method of R. C. offshore platform with the nonlinear finite element method proposed by the authors is reliable for practical use.