Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

In vivo astrocyte reprogramming following spinal cord injury

Harmful and helpful roles of astrocytes in spinal cord injury(SCI):SCI induce gradable sensory,motor and autonomic impairments that correlate with the lesion severity and the rostro-caudal location of the injury site.The absence of spontaneous axonal regeneration after injury results from neuron-intrinsic and neuron-extrinsic parameters.Indeed,not only adult neurons display limited capability to regrow axons but also the injury environment contains inhibitors to axonal regeneration and a lack of growth-promoting factors.Amongst other cell populations that respond to the lesion,reactive astrocytes were first considered as only detrimental to spontaneous axonal regeneration.Indeed,astrocytes.

Temperature-controlled DCP Fluorescent Probe Based on Hydrogel-immobilized Quantum Dots Composite

A composite was created by incorporating the quantum dot-enhanced SiO_(2)nanoparticles within this hydrogel.Based on this composite,a temperature-controlled fluorescent probe for DCP was developed.A meticulous examination of this probe revealed its attributes and factors affecting its performance.By using temperature modulation,the probe was adept at detecting DCP concentrations ranging between 1.0×10^(-6)and 9.0×10^(-6)mol/L.Such a probe offers remarkable selectivity,repeatability,and robust stability,so that the detection of DCP can be carried out at different temperatures,and a fast,reliable,sensitive and low-cost intelligent detection method is realized.

Transcriptional reprogramming during human osteoclast differentiation identifies regulators of osteoclast activity

Enhanced osteoclastogenesis and osteoclast activity contribute to the development of osteoporosis,which is characterized by increased bone resorption and inadequate bone formation.As novel antiosteoporotic therapeutics are needed,understanding the genetic regulation of human osteoclastogenesis could help identify potential treatment targets.This study aimed to provide an overview of transcriptional reprogramming during human osteoclast differentiation.Osteoclasts were differentiated from CD14+monocytes from eight female donors.RNA sequencing during differentiation revealed 8980 differentially expressed genes grouped into eight temporal patterns conserved across donors.These patterns revealed distinct molecular functions associated with postmenopausal osteoporosis susceptibility genes based on RNA from iliac crest biopsies and bone mineral density SNPs.Network analyses revealed mutual dependencies between temporal expression patterns and provided insight into subtype-specific transcriptional networks.The donor-specific expression patterns revealed genes at the monocyte stage,such as filamin B(FLNB)and oxidized low-density lipoprotein receptor 1(OLR1,encoding LOX-1),that are predictive of the resorptive activity of mature osteoclasts.The expression of differentially expressed G-protein coupled receptors was strong during osteoclast differentiation,and these receptors are associated with bone mineral density SNPs,suggesting that they play a pivotal role in osteoclast differentiation and activity.The regulatory effects of three differentially expressed G-protein coupled receptors were exemplified by in vitro pharmacological modulation of complement 5 A receptor 1(C5AR1),somatostatin receptor 2(SSTR2),and free fatty acid receptor 4(FFAR4/GPR120).Activating C5AR1 enhanced osteoclast formation,while activating SSTR2 decreased the resorptive activity of mature osteoclasts,and activating FFAR4 decreased both the number and resorptive activity of mature osteoclasts.In conclusion,we report the occurrence of transcriptional reprogramming during human osteoclast differentiation and identified SSTR2 and FFAR4 as antiresorptive G-protein coupled receptors and FLNB and LOX-1 as potential molecular markers of osteoclast activity.These data can help future investigations identify molecular regulators of osteoclast differentiation and activity and provide the basis for novel antiosteoporotic targets.

Glia-to-neuron reprogramming to the rescue?

Over the last two decades,the dogma that cell fate is immutable has been increasingly challenged,with important implications for regenerative medicine.The brea kth rough discovery that induced pluripotent stem cells could be generated from adult mouse fibroblasts is powerful proof that cell fate can be changed.An exciting extension of the discovery of cell fate impermanence is the direct cellular reprogram ming hypothesis-that terminally differentiated cells can be reprogrammed into other adult cell fates without first passing through a stem cell state.

Improved Unit Commitment with Accurate Dynamic Scenarios Clustering Based on Multi-Parametric Programming and Benders Decomposition

Stochastic unit commitment is one of the most powerful methods to address uncertainty. However, the existingscenario clustering technique for stochastic unit commitment cannot accurately select representative scenarios,which threatens the robustness of stochastic unit commitment and hinders its application. This paper providesa stochastic unit commitment with dynamic scenario clustering based on multi-parametric programming andBenders decomposition. The stochastic unit commitment is solved via the Benders decomposition, which decouplesthe primal problem into the master problem and two types of subproblems. In the master problem, the committedgenerator is determined, while the feasibility and optimality of generator output are checked in these twosubproblems. Scenarios are dynamically clustered during the subproblem solution process through the multiparametric programming with respect to the solution of the master problem. In other words, multiple scenariosare clustered into several representative scenarios after the subproblem is solved, and the Benders cut obtainedby the representative scenario is generated for the master problem. Different from the conventional stochasticunit commitment, the proposed approach integrates scenario clustering into the Benders decomposition solutionprocess. Such a clustering approach could accurately cluster representative scenarios that have impacts on theunit commitment. The proposed method is tested on a 6-bus system and the modified IEEE 118-bus system.Numerical results illustrate the effectiveness of the proposed method in clustering scenarios. Compared withthe conventional clustering method, the proposed method can accurately select representative scenarios whilemitigating computational burden, thus guaranteeing the robustness of unit commitment.

Combining reinforcement learning with mathematical programming:An approach for optimal design of heat exchanger networks

Heat integration is important for energy-saving in the process industry.It is linked to the persistently challenging task of optimal design of heat exchanger networks(HEN).Due to the inherent highly nonconvex nonlinear and combinatorial nature of the HEN problem,it is not easy to find solutions of high quality for large-scale problems.The reinforcement learning(RL)method,which learns strategies through ongoing exploration and exploitation,reveals advantages in such area.However,due to the complexity of the HEN design problem,the RL method for HEN should be dedicated and designed.A hybrid strategy combining RL with mathematical programming is proposed to take better advantage of both methods.An insightful state representation of the HEN structure as well as a customized reward function is introduced.A Q-learning algorithm is applied to update the HEN structure using theε-greedy strategy.Better results are obtained from three literature cases of different scales.

Uncertainty quantification of inverse analysis for geomaterials using probabilistic programming

Uncertainty is an essentially challenging for safe construction and long-term stability of geotechnical engineering.The inverse analysis is commonly utilized to determine the physico-mechanical parameters.However,conventional inverse analysis cannot deal with uncertainty in geotechnical and geological systems.In this study,a framework was developed to evaluate and quantify uncertainty in inverse analysis based on the reduced-order model(ROM)and probabilistic programming.The ROM was utilized to capture the mechanical and deformation properties of surrounding rock mass in geomechanical problems.Probabilistic programming was employed to evaluate uncertainty during construction in geotechnical engineering.A circular tunnel was then used to illustrate the proposed framework using analytical and numerical solution.The results show that the geomechanical parameters and associated uncertainty can be properly obtained and the proposed framework can capture the mechanical behaviors under uncertainty.Then,a slope case was employed to demonstrate the performance of the developed framework.The results prove that the proposed framework provides a scientific,feasible,and effective tool to characterize the properties and physical mechanism of geomaterials under uncertainty in geotechnical engineering problems.

Rapid Prototype Development Approach for Genetic Programming

Genetic Programming (GP) is an important approach to deal with complex problem analysis and modeling, and has been applied in a wide range of areas. The development of GP involves various aspects, including design of genetic operators, evolutionary controls and implementations of heuristic strategy, evaluations and other mechanisms. When designing genetic operators, it is necessary to consider the possible limitations of encoding methods of individuals. And when selecting evolutionary control strategies, it is also necessary to balance search efficiency and diversity based on representation characteristics as well as the problem itself. More importantly, all of these matters, among others, have to be implemented through tedious coding work. Therefore, GP development is both complex and time-consuming. To overcome some of these difficulties that hinder the enhancement of GP development efficiency, we explore the feasibility of mutual assistance among GP variants, and then propose a rapid GP prototyping development method based on πGrammatical Evolution (πGE). It is demonstrated through regression analysis experiments that not only is this method beneficial for the GP developers to get rid of some tedious implementations, but also enables them to concentrate on the essence of the referred problem, such as individual representation, decoding means and evaluation. Additionally, it provides new insights into the roles of individual delineations in phenotypes and semantic research of individuals.

Sb掺杂Li_(2)SnO_(3)增加氧化电势提升2,4-DCP光催化降解的研究

本文通过固相法制备Sb掺杂Li_(2)SnO_(3)光催化剂并将其用于光催化降解2,4-DCP。通过粉末X射线多晶衍射技术(PXRD)、高分辨透射电镜(TEM)及扫描电镜(SEM)对合成的光催化剂分别进行了物相结构和表面形貌表征。光催化实验表明,Sb掺杂之后的样品具有较为优越的光催化性能,其在紫外光下12 min内,其降解率可达100%。然而母体Li_(2)SnO_(3)在同样的条件下,12 min内光催化降解率仅有58.20%。紫外漫反射吸收光谱表明,Sb掺杂之后,光学吸收带宽增加;时间分辨光谱测试表明,Sb掺杂后样品的光生载流子寿命下降。综合以上分析,光催化效率提升的原因,来自于其价带氧化电势的增强。自由基捕获实验表明,空穴(h+)和超氧阴离子自由基(O_(2)·-)是2,4-DCP降解过程中的主要活性自由基。本文的研究为半导体掺杂对光催化效率的提升提供重要的指导。

Adaptive Optimal Discrete-Time Output-Feedback Using an Internal Model Principle and Adaptive Dynamic Programming

In order to address the output feedback issue for linear discrete-time systems, this work suggests a brand-new adaptive dynamic programming(ADP) technique based on the internal model principle(IMP). The proposed method, termed as IMP-ADP, does not require complete state feedback-merely the measurement of input and output data. More specifically, based on the IMP, the output control problem can first be converted into a stabilization problem. We then design an observer to reproduce the full state of the system by measuring the inputs and outputs. Moreover, this technique includes both a policy iteration algorithm and a value iteration algorithm to determine the optimal feedback gain without using a dynamic system model. It is important that with this concept one does not need to solve the regulator equation. Finally, this control method was tested on an inverter system of grid-connected LCLs to demonstrate that the proposed method provides the desired performance in terms of both tracking and disturbance rejection.

路基压实质量检测中DCP法的适用性分析

为有效提升路基施工质量,保证道路安全稳定运营,加强路基压实质量检测尤为重要。鉴于此,针对路基压实质量检测DCP法适用性进行研究,阐述了DCP法基本原理及现场检测步骤,根据现场DCP及CBR检测结果实施相关性分析,验证了贯入度与CBR值之间的线性关系,确定了DCP法工程应用范围,可供路基质量检测人员参考。

A Dimensional Reduction Approach Based on Essential Constraints in Linear Programming

This paper presents a new dimension reduction strategy for medium and large-scale linear programming problems. The proposed method uses a subset of the original constraints and combines two algorithms: the weighted average and the cosine simplex algorithm. The first approach identifies binding constraints by using the weighted average of each constraint, whereas the second algorithm is based on the cosine similarity between the vector of the objective function and the constraints. These two approaches are complementary, and when used together, they locate the essential subset of initial constraints required for solving medium and large-scale linear programming problems. After reducing the dimension of the linear programming problem using the subset of the essential constraints, the solution method can be chosen from any suitable method for linear programming. The proposed approach was applied to a set of well-known benchmarks as well as more than 2000 random medium and large-scale linear programming problems. The results are promising, indicating that the new approach contributes to the reduction of both the size of the problems and the total number of iterations required. A tree-based classification model also confirmed the need for combining the two approaches. A detailed numerical example, the general numerical results, and the statistical analysis for the decision tree procedure are presented.

Optimization Techniques for GPU-Based Parallel Programming Models in High-Performance

This study embarks on a comprehensive examination of optimization techniques within GPU-based parallel programming models,pivotal for advancing high-performance computing(HPC).Emphasizing the transition of GPUs from graphic-centric processors to versatile computing units,it delves into the nuanced optimization of memory access,thread management,algorithmic design,and data structures.These optimizations are critical for exploiting the parallel processing capabilities of GPUs,addressingboth the theoretical frameworks and practical implementations.By integrating advanced strategies such as memory coalescing,dynamic scheduling,and parallel algorithmic transformations,this research aims to significantly elevate computational efficiency and throughput.The findings underscore the potential of optimized GPU programming to revolutionize computational tasks across various domains,highlighting a pathway towards achieving unparalleled processing power and efficiency in HPC environments.The paper not only contributes to the academic discourse on GPU optimization but also provides actionable insights for developers,fostering advancements in computational sciences and technology.

PROFINET IO设备中DCP协议的研究与应用

为解决自动化外围现场设备接入PROFINET网络,而同类设备依赖特定芯片,成本较高,可移植性差的问题,利用软件设计实现普通网卡下的PROFINET IO通信;在树莓派3B+开发板上实现IO设备,支持非同步实时数据交换,可以通过槽/子槽数据模型匹配IO模块;通过设计描述性文件,使其IO模块类型映射模块、子模块的ID,实现模块化IO设备,允许根据需要添加和删除模块;系统启动时控制器根据设备名称寻址,再基于设备名称为IO设备分配IP地址,这一过程的实现依赖DCP协议,它是PROFINET网络中数据交互的基础;为了深入探究PROFINET协议栈,在实现IO通信的基础上从DCP协议的原理入手,对其功能和程序实现进行了详细研究,并将所实现的IO设备与PLC S7-1200进行组态,验证了IO设备所实现的功能和DCP工作的过程;实验测试表明,树莓派实现了PROFINET RT和NRT通信功能,其更新时间抖动小于50μs,满足控制工程上的应用要求。

基于DCP-Imp CycleGAN CenterNet去雾算法的交通标志检测

针对雾天环境下对小型交通标志检测效果不佳的问题,提出了一种基于DCP-Imp CycleGAN与CenterNet融合去雾的交通标志检测方法。该方法在预处理模块将DCP算法嵌入到优化后的CycleGAN网络框架中,对图像高质量的纹理信息细化处理,再通过感知融合模块,获得更易被识别且自然的无雾图像;为了进一步提高对小型交通标志的识别能力,对CenterNet中的残差块进行了轻量化设计,同时引入了CBAM注意力机制和FPN特征融合模块,进而减少了有效特征信息的丢失。实验结果表明,改进算法能有效解决图像去雾中色差明显和不清晰的问题,在CCTSDB数据集上实验评估得到的mAP较CenterNet提升了5.48%,FPS提升了4帧,有效解决了雾天环境下对小型交通标志的漏检、误检问题。

基于DCP的不同固化粉土基层路用性能分析

为分析沥青乳液、沥青微粉、无机复合型三种不同固化粉土基层的路用性能,依托黄河冲积平原农村公路项目进行不同结构层的现场动力锥贯入仪(DCP)试验,同时与8%水泥固化粉土基层对比分析。结果表明:沥青乳液、沥青微粉、无机复合型固化粉土基层的贯入度D_(d)为1.33~1.74 mm/blow,均小于水泥固化粉土结构层的贯入度2.09 mm/blow,其整体固化效果更优。三种不同固化粉土的DCP换算无侧限抗压强度与取芯实测无侧限抗压强度、DCP累积平均贯入度与FWD弯沉之间具有良好的相关性,采用分层计算与累积计算可分别用于单层强度和整体承载能力的快速评价。DCP测试结果还可为黄河冲积平原粉土改良固化路用性能的评价提供数据参考与工程借鉴。

天然气管道压缩机组DCP/UPS电源的安全环保节能改造

我国天然气管道初期工程中大量采用国外先进压缩机技术,由于其采用国外标准和根据国外供电条件设计,在我国的实际应用中产生很多问题,供电问题是其中的一种。文章基于目前天然气压气站供电状况存在的问题,提出了采用国产电源替代的节能改造方案。改造前的DCP/UPS电源采用相控模式,效率只有70%左右,改造后效率可提高至92%。DCP/UPS电源改造已在西气东输、西部管道及陕京线成功改造了50多套。实践证明:用国产DCP/UPS电源替代,不仅设计合理、运行可靠,维护方便,而且备件充足、成本低。

基于互联网的DCP端到端自动分发方法与实现

随着我国互联网的发展以及近年来疫情的影响,通过网络进行影片分发的优势逐渐显现。同时,基于当前信息化管理、智能化升级总体发展方向,面对我国影院放映模式多样化发展趋势,本文对当前主流电影分发模式进行分析,提出并实现了一种基于互联网的数字电影数据包(DCP)端到端自动分发方法,实现以影片制作方为起点,以影片放映方为终点,以影片分发方为整体控制端,对影片DCP从分发准备到结束的全流程控制与信息管理,尽可能简化了电影拷贝过程中的人工干预,确保分发影片文件的安全性和完整性,可满足多种电影制作发行放映体系,保证各类影院影片放映的时效性。

2,4-DCP对若蚓期、成蚓期赤子爱胜蚓毒性效应研究

广泛使用的2,4-二氯酚(2,4-DCP)是一种常见的难降解有机物,在土壤中有一定的残留。为研究其急性毒性效应,分别以生长阶段为若蚓期、成蚓期的赤子爱胜蚓(Eisenia fetida)为受试生物,采用滤纸接触法和两室趋避试验探讨不同暴露剂量2,4-DCP对蚯蚓的急性毒性响应。结果表明,滤纸法测得若蚓期蚯蚓24 h、48 h和72 h-LC 50分别为80.71、76.20、73.64μg·cm^(-2),而成蚓期蚯蚓的分别为82.88、77.63、74.65μg·cm^(-2),略高于若蚓期;蚯蚓的中毒现象表现为部分环节肿大并渗血、躯体颜色变透明、躯体萎缩与断节等。两室趋避试验表明,当土壤中2,4-DCP浓度为10 mg·kg^(-1)时,清洁土壤中蚯蚓占比高于总数的80%,即表现出明显的趋避行为。蚯蚓组织切片显微观察结果表明,70.74μg·cm^(-2)浓度暴露下蚯蚓表皮结构受损,黄色细胞破裂,而88.42μg·cm^(-2)浓度暴露下蚯蚓表皮严重溃烂坏死,纵肌细胞间隙扩大,黄色细胞和肠上皮细胞发生破裂、变性并与肠壁分离;滤纸法暴露24 h时,蚯蚓体内蛋白质含量、SOD活性和CAT活性随2,4-DCP暴露剂量呈现先上升后降低趋势,表现出较明显的低剂量刺激和高剂量抑制效应;低浓度2,4-DCP胁迫下,若蚓体内的蛋白质含量、SOD活性和CAT活性显著高于成蚓;长时间高浓度暴露下的若蚓体内SOD活性、CAT活性几乎均低于成蚓。蚯蚓在若蚓期、成蚓期这2个生长阶段对2,4-DCP表现出不同的应激效应,其耐受能力也存在差异。