Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Use of Ligament Advanced Reinforcement System tube in stabilization of proximal humeral endoprostheses

AIM:To review outcomes following usage of the Ligament Advanced Reinforcement System(LARS?)in shoulder tumors.METHODS:Medical records of nineteen patients(19 shoulders)that underwent tumor excisional procedure and reconstruction with the LARS synthetic fabric,were retrospectively reviewed.RESULTS:Patients’median age was 58 years old,while the median length of resection was 110 mm(range 60-210 mm).Compared to immediate post-operative radiographs,the prosthesis mean end-point position migrated superiorly at a mean follow up period of 26 mo(P=0.002).No statistical significant correlations between the prosthesis head size(P=0.87);the implant stem body length(P=0.949);and the length of resection(P=0.125)with the position of the head,were found at last follow up.Two cases of radiological dislocation were noted but only one was clinically symptomatic.A minor superficial wound dehiscence,healed without surgery,occurred.There was no evidence of aseptic loosening either,and no prosthetic failure.CONCLUSION:LARS?use ensured stability of the shoulder following endoprosthetic reconstruction in most patients.

Arthroscopic anatomical reconstruction of lateral collateral ligaments with ligament advanced reinforcement system artificial ligament for chronic ankle instability

BACKGROUND Recently,the use of ligament advanced reinforcement system(LARS)artificial ligament,a new graft which has several unique advantages such as no donor-site morbidity,early recovery and no risk of disease transmission which has been a significant breakthrough for anatomical ligament reconstruction.Growing studies suggested that the special design of the LARS ligament with open fibers in its intra-articular part was believed to be more resistant to torsional fatigue and wearing.However,the safety and efficacy of LARS artificial ligament for ankle joint lateral collateral ankle ligament reconstruction has not been defined to date.AIM To evaluate the clinical results of all-arthroscopic anatomical reconstruction of ankle joint lateral collateral ligaments with the LARS artificial ligament for chronic ankle instability.METHODS Twenty-two patients with chronic lateral instability underwent anatomical reconstruction of the lateral collateral ligaments of ankle with LARS artificial ligament.The visual analogue score(VAS),American Orthopaedic Foot and Ankle Society score(AOFAS score)and Karlsson score were used to evaluate the clinical results before and after surgery.RESULTS A total of 22 patients(22 ankles)were followed up for a mean of 12 mo.All patients reported significant improvement compared to their preoperative status.The mean AOFAS score improved from 42.3±4.9 preoperatively to 90.4±6.7 postoperatively.The mean Karlsson score improved from 38.5±3.2 preoperatively to 90.1±7.8 postoperatively.The mean VAS score improved from 1.9±2.5 preoperatively to 0.8±1.7 postoperatively.CONCLUSION All-arthroscopic anatomical reconstruction of the lateral collateral ligaments with LARS artificial ligament achieved a satisfactory surgical outcome for chronic ankle instability.

Nonlinear stability of advanced sandwich cylindrical shells comprising porous functionally graded material and carbon nanotube reinforced composite layers under elevated temperature

The nonlinear stability of sandwich cylindrical shells comprising porous functionally graded material(FGM) and carbon nanotube reinforced composite(CNTRC)layers subjected to uniform temperature rise is investigated. Two sandwich models corresponding to CNTRC and FGM face sheets are proposed. Carbon nanotubes(CNTs) in the CNTRC layer are embedded into a matrix according to functionally graded distributions. The effects of porosity in the FGM and the temperature dependence of properties of all constituent materials are considered. The effective properties of the porous FGM and CNTRC are determined by using the modified and extended versions of a linear mixture rule, respectively. The basic equations governing the stability problem of thin sandwich cylindrical shells are established within the framework of the Donnell shell theory including the von K’arm’an-Donnell nonlinearity. These equations are solved by using the multi-term analytical solutions and the Galerkin method for simply supported shells.The critical buckling temperatures and postbuckling paths are determined through an iteration procedure. The study reveals that the sandwich shell model with a CNTRC core layer and relatively thin porous FGM face sheets can have the best capacity of thermal load carrying. In addition, unlike the cases of mechanical loads, porosities have beneficial effects on the nonlinear stability of sandwich shells under the thermal load. It is suggested that an appropriate combination of advantages of FGM and CNTRC can result in optimal efficiency for advanced sandwich structures.

基于Advanced-TD3算法的卫星探索控制策略

卫星控制算法在卫星控制领域拥有十分重要的地位,而深度强化学习则是当前前沿的卫星控制算法之一。针对目前太空环境日渐复杂的问题,提出了基于TD3算法的改进TD3(Advanced-TD3)算法,实现控制卫星到达预定目标区域。在开源环境中进行仿真实验,实验结果验证了该算法的空间探索能力,拥有较高的鲁棒性,可以较为精确地帮助卫星完成控制问题,增强卫星对复杂空间中的控制能力,提高卫星的运行效率。

A reinforcement learning approach to vehicle coordination for structured advanced air mobility

Advanced Air Mobility(AAM)has emerged as a pioneering concept designed to optimize the efficacy and ecological sustainability of air transportation.Its core objective is to provide highly automated air transportation services for passengers or cargo,operating at low altitudes within urban,suburban,and rural regions.AAM seeks to enhance the efficiency and environmental viability of the aviation sector by revolutionizing the way air travel is conducted.In a complex aviation environment,traffic management and control are essential technologies for safe and effective AAM operations.One of the most difficult obstacles in the envisioned AAM systems is vehicle coordination at merging points and intersections.The escalating demand for air mobility services,particularly within urban areas,poses significant complexities to the execution of such missions.In this study,we propose a novel multi-agent reinforcement learning(MARL)approach to efficiently manage high-density AAM operations in structured airspace.Our approach provides effective guidance to AAM vehicles,ensuring conflict avoidance,mitigating traffic congestion,reducing travel time,and maintaining safe separation.Specifically,intelligent learning-based algorithms are developed to provide speed guidance for each AAM vehicle,ensuring secure merging into air corridors and safe passage through intersections.To validate the effectiveness of our proposed model,we conduct training and evaluation using BlueSky,an open-source air traffic control simulation environment.Through the simulation of thousands of aircraft and the integration of real-world data,our study demonstrates the promising potential of MARL in enabling safe and efficient AAM operations.The simulation results validate the efficacy of our approach and its ability to achieve the desired outcomes.

堆积体隧道超前加固方式研究

为探究堆积体隧道超前加固方式,建立了软弱堆积体隧道三维计算模型,通过数值模拟对比了不同超前加固方式下围岩隧道变形与结构受力情况,确定了堆积体隧道最佳超前加固方式为帷幕注浆+大管棚。主要研究结论如下:双层小导管方案相比大管棚方案可控制拱顶部位的塑性区与沉降变形,减小结构拉应力,但有增大结构受压破坏的风险;双层小导管与大管棚方案相比帷幕注浆方案无法有效控制围岩变形,对堆积体隧道适用性较差;帷幕注浆方案可大幅减小围岩变形以及左右拱肩塑性区,抑制隧道与支护结构变形,控制围岩压应力与衬砌受力;帷幕注浆+大管棚方案相比帷幕注浆+单层小导管方案可进一步提升整体围岩变形与支护结构受力。

Characteristics of bone tunnel changes after anterior cruciate ligament reconstruction using Ligament Advanced Reinforcement System artificial ligament

Background There are different materials used for anterior cruciate ligament （ACL） reconstruction. It has been reported that both autologous grafts and allografts used in ACL reconstruction can cause bone tunnel enlargement. This study aimed to observe the characteristics of bone tunnel changes and possible causative factors following ACL reconstruction using Ligament Advanced Reinforcement System （LARS） artificial ligament. Methods Forty-three patients underwent ACL reconstruction using LARS artificial ligament and were followed up for 3 years. X-ray and CT examinations were performed at 1,3, 6, 12, 24, and 36 months after surgery, to measure the width of tibial and femoral tunnels. Knee function was evaluated according to the Lysholm scoring system. The anterior and posterior stability of the knee was measured using the KT-1000 arthrometer. Results According to the Peyrache grading method, grade 1 femoral bone tunnel enlargement was observed in three cases six months after surgery. No grade 2 or grade 3 bone tunnel enlargement was found. The bone tunnel enlargement in the three cases was close to the articular surface with an average tunnel enlargement of （2.5＋0.3） mm. Forty cases were evaluated as grade 0. The average tibial and femoral tunnel enlargements at the last follow-up were （0.8＋0.3） and （1.1＋0.3） mm, respectively. There was no statistically significant difference in bone tunnel width changes at different time points （P 〉0.05）. X-ray and CT measurements were consistent. Conclusions There was no marked bone tunnel enlargement immediately following ACL reconstruction using LARS artificial ligament. Such enlargement may, however, result from varying grafting factors involving the LARS artificial ligament or from different fixation methods.

Traffic signal control in mixed traffic environment based on advance decision and reinforcement learning

Reinforcement learning-based traffic signal control systems (RLTSC) can enhance dynamic adaptability, save vehicle travelling timeand promote intersection capacity. However, the existing RLTSC methods do not consider the driver’s response time requirement, sothe systems often face efficiency limitations and implementation difficulties.We propose the advance decision-making reinforcementlearning traffic signal control (AD-RLTSC) algorithm to improve traffic efficiency while ensuring safety in mixed traffic environment.First, the relationship between the intersection perception range and the signal control period is established and the trust region state(TRS) is proposed. Then, the scalable state matrix is dynamically adjusted to decide the future signal light status. The decision will bedisplayed to the human-driven vehicles (HDVs) through the bi-countdown timer mechanism and sent to the nearby connected automatedvehicles (CAVs) using the wireless network rather than be executed immediately. HDVs and CAVs optimize the driving speedbased on the remaining green (or red) time. Besides, the Double Dueling Deep Q-learning Network algorithm is used for reinforcementlearning training;a standardized reward is proposed to enhance the performance of intersection control and prioritized experiencereplay is adopted to improve sample utilization. The experimental results on vehicle micro-behaviour and traffic macro-efficiencyshowed that the proposed AD-RLTSC algorithm can simultaneously improve both traffic efficiency and traffic flow stability.

综采面沿空巷道超前动压区煤柱失稳效应与锚注加固技术

为探究综采面沿空巷道超前动压区煤柱对巷道围岩整体稳定的影响规律,综合采用数值模拟、工程应用等方法,建立沿空巷道超前巷道数值分析模型,研究综采面回采期间不同煤柱宽度下沿空巷道围岩力学和变形响应特征。研究结果表明:护巷煤柱宽度小于5 m时,工作面帮为采动应力主要承载结构,其内部应力及变形均大于煤柱帮,此时工作面帮应作为超前动压区重点加固对象,随着护巷煤柱宽度增加,煤柱承载应力逐渐增加、变形增大,护巷煤柱宽度增加至5 m以上时,煤柱承载应力显著增加,采动应力由沿空巷道两帮围岩共同承载,超前动压区应同时加强两帮支护,基于此开发超前动压区沿空巷道锚注加固技术,并进行工程应用。研究结果可为平顶山矿区类似条件沿空巷道超前控制提供一定参考。

基于数字孪生和深度强化学习的矿井超前液压支架自适应抗冲支护方法

受深部开采冲击地压等地质灾害扰动的影响,存在矿井超前支护系统自感知能力差、智能抗冲自适应能力弱、缺乏决策控制能力等问题。针对上述问题,提出了一种基于数字孪生和深度强化学习的矿井超前液压支架自适应抗冲支护方法。通过多源传感器感知巷道环境和超前液压支架支护状态,在虚拟世界中创建物理实体的数字孪生模型,其中物理模型精确展现超前液压支架的结构特征和细节,控制模型实现超前液压支架的自适应控制,机理模型实现对超前液压支架自适应支护的逻辑描述和机理解释,数据模型存储超前液压支架实体运行数据和孪生数据,仿真模型完成超前液压支架立柱仿真以实现超前液压支架与数字孪生模型虚实交互。根据基于深度Q网络(DQN)的超前液压支架自适应抗冲决策算法,对仿真环境中巷道抗冲支护进行智能决策,并依据决策结果对物理实体和数字孪生模型下达调控指令,实现超前液压支架智能控制。实验结果表明:立柱位移与压力变化一致,说明超前液压支架立柱仿真模型设计合理,从而验证了数字孪生模型的准确性;基于DQN的矿井超前液压支架自适应抗冲决策算法可通过调节液压支架控制器PID参数,自适应调控立柱压力,提升巷道安全等级,实现超前液压支架自适应抗冲支护。

面向中外学生的混凝土结构教学实践与思考

该文分析“一带一路”倡议背景下留学生混凝土结构教学面临的国际化教学挑战,传统的混凝土教学模式在国外留学生加入教育背景下亟需改进。该文首先分析天津大学土木工程系留学生构成比例及混凝土教学所面临的不同国家混凝土设计规范差异,分析作者所教授的混凝土结构设计和全英文高等混凝土结构等课程特点,并提出加强课程思政、丰富教学方法、重视实习基地、因材施教调整教学内容和多重考核方式的课程教学措施。以期达到提升混凝土教学质量,提升中外学生培养质量,增强中外学生人才培养素质,助力我国“一带一路”发展建设。

LARS肿瘤管在肿瘤型人工髋关节置换术关节功能重建中的临床应用

目的:股骨近端肿瘤切除常导致髋关节不稳定和功能缺失,临床上已有多种方法用于修复髋关节软组织功能,但仍存在不足。本研究旨在评价韧带高级加固系统(ligament advanced reinforcement system,LARS)肿瘤管对辅助肿瘤型人工髋关节置换术患者软组织功能重建的优势与不足。方法:2012年1月至2022年1月本研究共纳入湘雅骨肿瘤治疗中心的股骨近端肿瘤患者85例(男41例,女44例),年龄10~79(38.5±18.2)岁,其中良性侵袭性肿瘤13例,原发恶性骨肿瘤45例,骨转移瘤27例。收集患者的临床资料、影像学资料和术中照片,对患者进行随访和术后功能评价。分别采用肌肉骨骼肿瘤学会(Musculoskeletal Tumor Society,MSTS)评分系统和Harris髋关节评分系统评价患者的肢体功能和髋关节功能。结果:37例(43.5%)患者术前合并病理性骨折,病灶长度为(9.4±2.9)cm。非转移瘤患者术后复发7例,其中骨肉瘤6例,纤维肉瘤1例。9例骨肉瘤患者出现肺转移。5例因术后并发症再次手术,其中3例为深静脉血栓形成,1例为巨细胞肉芽肿,1例为假体周围感染。5例术后出现Trendelenburg步态。6例术后出现双下肢不等长。患者术后MSTS评分为26.7±1.4,Harris评分为89.6±5.3。结论:LARS肿瘤管可以有效辅助重建患者髋关节的软组织功能,并极大地减少了术后并发症的发生,是肿瘤型人工髋关节置换术中关节功能重建的有效技术改良。

陈四楼煤矿地面定向钻孔超前区域治理底板岩溶水害技术

为了进一步提高极复杂地质构造下矿井底板岩溶水害区域治理技术的有效性,降低底板突水的危险性,解决矿井区域水害治理过程中面临的太原组上段灰岩含水层岩溶裂隙发育、水压高、富水性强、井下钻孔工程量大、施工工期长等安全技术难题,通过分析矿井近年来区域治理后工作面底板出水案例及特征,提出了地面定向钻孔超前区域治理底板岩溶水害技术。以陈四楼煤矿南翼九采区2901、2903试验工作面开采为背景,选取了L8灰岩含水层作为目标层位,设计了地面定向孔组的技术参数,并通过现场试验测试分析了治理前后的二2煤底板水压和各钻孔的吸水率。结果表明:注浆后各钻孔的吸水率呈现明显的下降趋势,Z2和Z3孔组钻孔平均吸水率分别下降97%和94%,经过定向钻孔超前注浆加固底板太原组上段灰岩后,岩溶裂隙得到了有效充填;治理后区内太原组上段L8灰岩内水压下降至1.2~4.2 MPa,且水压分布不再与地层埋深吻合,多呈孤岛状,太原组上段L8灰被改造为等效隔水层,底板突水系数下降至0.023~0.057 MPa/m。技术成功应用实现了极复杂地质构造下强突水危险性工作面的安全高效回采,为同类矿井相似型综采面的底板岩溶水害防治,探索出了一套新的方法。

水-岩软化作用下巴东组泥岩隧道大断面开挖掌子面稳定性研究

为明确软弱围岩条件下高速铁路隧道大断面机械化施工过程中的掌子面稳定性特征,以郑万铁路巴东隧道穿越巴东组紫红泥岩段为工程背景,对隧道揭露的巴东组紫红泥岩开展不同含水率下的三轴试验,揭示巴东隧道紫红泥岩的水-岩软化特征;在考虑水-岩软化特征的Hoek-Brown强度准则下,构建考虑水-岩软化作用的隧道穿越巴东紫红泥岩地层流-固耦合数值模型,揭示不同地层水位下巴东组紫红泥岩隧道大断面施工掌子面稳定性特征;最终提出“掌子面喷混凝土+掌子面超前锚杆+超前管棚”的组合式超前加固方案。经过试验和模拟计算,结果表明:1)随含水量的增加,巴东隧道紫红泥岩的刚度与抗剪强度显著降低,与既有研究揭示的浅表地层紫红泥岩相比,巴东隧道紫红泥岩具有更高的抗剪强度;2)掌子面后方拱顶沉降、掌子面前方拱顶预收敛、掌子面挤出变形的最大值均随历史水位的升高而增大;3)根据不同地层历史水位条件下开挖面支护力与挤出变形之间的关系,巴东隧道紫红泥岩掌子面稳定性可以分为A(稳定)、B(短期稳定)以及C(不稳定)3类,对于C类掌子面,采用“掌子面喷混凝土+掌子面超前锚杆+超前管棚”的组合式超前加固措施后,掌子面前后方各项变形最大值可以减小60%~90%。

东滩煤矿3308综放工作面沿空巷道治理及支护技术研究

东滩煤矿3308综放工作面轨道顺槽为沿采空区掘进巷道,与原3307工作面运输顺槽临近,中间预留煤柱约3.5 m。回采期间,受超前动压及沿空侧向支承压力影响,轨道顺槽围岩变形较大。为解决回采期间轨道顺槽顶板下沉、鼓帮影响支架拉移等问题,采取巷道顶板补打29U锚索梁、帮部补打锚索、钢带等方式增加巷道主动、被动支护强度,控制围岩变形,保证轨道顺槽支架正常拉移空间及人员作业空间,保障了井下生产和作业人员安全。

超前注浆加固技术在掘进工作面过破碎带中的应用

G煤矿在回采工作面5521-10回风巷掘进期间,受到褶曲、断层等地质构造及煤层赋存条件的影响,构造前后形成集中应力区,煤岩体处于松软破碎状态,原有锚网支护方式无法有效满足巷道围岩控制要求,巷道掘进期间易发生漏顶造成灾害。针对上述问题,结合掘进工作面实际情况,提出了解决思路,决定采用尤卡尼加固剂进行巷道煤岩体超前注浆加固的技术措施,并设计了相应的技术参数,注浆加固后使破碎煤岩体力学性质得到了有效改善;注浆加固技术方案实施后,选择合理的支护参数对巷道进行有效支护,并对巷道进行离层变形监测、锚杆锚索受力特征现场测试,其结果验证了该技术的合理性。

和尚嘴矿煤层胶带大巷掘进与支护技术探讨

和尚嘴煤矿14号煤层胶带大巷掘进过程中的主要风险为上部小窑旧巷;为保障该巷道安全掘进,拟定采用物探结合钻探的超前勘探、采取对小窑旧巷的区域注浆填充、增加U型钢支护等技术措施;对该大巷235~260 m区间的上部旧巷实施了上述治理措施后,掘进已安全通过该区域;应用表明,大巷支护效果良好。

复杂条件下盾构隧道长距离穿越危桥关键控制技术

乌鲁木齐市某地铁区间盾构隧道长距离顺穿人民路立交桥及河滩快速路,人民路立交桥评定为危桥。以隧道内盾构掘进过程中盾构参数、出渣量、同步注浆、二次注浆为主要技术控制措施,以隧道外地质超前预报、土体预加固、桥梁预支顶、自动化监控量测为辅助技术措施进行控制,分析不同穿越阶段桥梁及路面变形。分析结果显示,所采取的技术措施能有效控制盾构掘进不同阶段对建筑物及地面的影响。

隧道浅埋段穿越软弱复杂地层加固处理技术

某隧道进口浅埋段穿越风化砂、细角砾土和碎石土层,局部夹有软粉质黏土和块石,围岩自稳能力差,隧底局部承载力不足。介绍穿越软弱复杂地层加固动态处理技术,通过采用地表注浆加固、洞外增设第2层超长管棚、优化CRD工法等措施,实施隧道超前周边注浆、隧底钢管桩注浆等加固处理技术,确保施工安全和结构安全。