Path Planning and Tracking Control for Parking via Soft Actor-Critic Under Non-Ideal Scenarios

Parking in a small parking lot within limited space poses a difficult task. It often leads to deviations between the final parking posture and the target posture. These deviations can lead to partial occupancy of adjacent parking lots, which poses a safety threat to vehicles parked in these parking lots. However, previous studies have not addressed this issue. In this paper, we aim to evaluate the impact of parking deviation of existing vehicles next to the target parking lot(PDEVNTPL) on the automatic ego vehicle(AEV) parking, in terms of safety, comfort, accuracy, and efficiency of parking. A segmented parking training framework(SPTF) based on soft actor-critic(SAC) is proposed to improve parking performance. In the proposed method, the SAC algorithm incorporates strategy entropy into the objective function, to enable the AEV to learn parking strategies based on a more comprehensive understanding of the environment. Additionally, the SPTF simplifies complex parking tasks to maintain the high performance of deep reinforcement learning(DRL). The experimental results reveal that the PDEVNTPL has a detrimental influence on the AEV parking in terms of safety, accuracy, and comfort, leading to reductions of more than 27%, 54%, and 26%respectively. However, the SAC-based SPTF effectively mitigates this impact, resulting in a considerable increase in the parking success rate from 71% to 93%. Furthermore, the heading angle deviation is significantly reduced from 2.25 degrees to 0.43degrees.

融合Dead-ends和离线监督Actor-Critic的动态治疗策略生成模型

强化学习对数学模型依赖性低,利用经验便于架构和优化模型,非常适合用于动态治疗策略学习。但现有研究仍存在以下问题:1)学习策略最优性的同时未考虑风险,导致学到的策略存在一定的风险;2)忽略了分布偏移问题,导致学到的策略与医生策略完全不同;3)忽略患者的历史观测数据和治疗史,从而不能很好地得到患者状态,进而导致不能学到最优策略。基于此,提出了融合Dead-ends和离线监督Actor-Critic的动态治疗策略生成模型DOSAC-DTR。首先,考虑学到的策略所推荐的治疗行动的风险性,在Actor-Critic框架中融入Dead-ends概念;其次,为缓解分布偏移问题,在Actor-Critic框架中融入医生监督,在最大化预期回报的同时,最小化所学策略与医生策略之间的差距;最后,为了得到包含患者关键历史信息的状态表示,使用基于LSTM的编码器解码器模型对患者的历史观测数据和治疗史进行建模。实验结果表明,DOSAC-DTR相比基线方法有更好的性能,可以得到更低的估计死亡率以及更高的Jaccard系数。

无人机辅助物联网中基于Safe Actor-Critic的信息年龄最小化研究

无人机作为一种新的通信设备,有望在物联网数据采集、监控等业务中发挥关键作用。为保证所采集数据的时效性,利用信息年龄来衡量无人机从物联网设备接收到的数据新鲜度。通过联合优化无人机轨迹和无人机与物联网设备的关联策略以最小化信息年龄加权和,并保证无人机累积飞行能量消耗满足预算要求。由于上述问题同时受短期和长期约束条件的限制,将问题建模为受约束的马尔可夫决策过程(CMDP),并利用Safe Actor-Critic来求解。仿真结果表明,所提算法在最小化信息年龄的同时,能有效保证能量预算。

基于不确定性估计的离线确定型Actor-Critic

Actor-Critic是一种强化学习方法,通过与环境在线试错交互收集样本来学习策略,是求解序贯感知决策问题的有效手段.但是,这种在线交互的主动学习范式在一些复杂真实环境中收集样本时会带来成本和安全问题离线强化学习作为一种基于数据驱动的强化学习范式,强调从静态样本数据集中学习策略,与环境无探索交互,为机器人、自动驾驶、健康护理等真实世界部署应用提供了可行的解决方案,是近年来的研究热点.目前,离线强化学习方法存在学习策略和行为策略之间的分布偏移挑战,针对这个挑战,通常采用策略约束或值函数正则化来限制访问数据集分布之外(Out-Of-Distribution,OOD)的动作,从而导致学习性能过于保守,阻碍了值函数网络的泛化和学习策略的性能提升.为此,本文利用不确定性估计和OOD采样来平衡值函数学习的泛化性和保守性,提出一种基于不确定性估计的离线确定型Actor-Critic方法(Offline Deterministic Actor-Critic based on UncertaintyEstimation,ODACUE).首先,针对确定型策略,给出一种Q值函数的不确定性估计算子定义,理论证明了该算子学到的Q值函数是最优Q值函数的一种悲观估计.然后,将不确定性估计算子应用于确定型Actor-Critic框架中,通过对不确定性估计算子进行凸组合构造Critic学习的目标函数.最后,D4RL基准数据集任务上的实验结果表明:相较于对比算法,ODACUE在11个不同质量等级数据集任务中的总体性能提升最低达9.56%,最高达64.92%.此外,参数分析和消融实验进一步验证了ODACUE的稳定性和泛化能力.

GRU-integrated constrained soft actor-critic learning enabled fully distributed scheduling strategy for residential virtual power plant

In this study,a novel residential virtual power plant(RVPP)scheduling method that leverages a gate recurrent unit(GRU)-integrated deep reinforcement learning(DRL)algorithm is proposed.In the proposed scheme,the GRU-integrated DRL algorithm guides the RVPP to participate effectively in both the day-ahead and real-time markets,lowering the electricity purchase costs and consumption risks for end-users.The Lagrangian relaxation technique is introduced to transform the constrained Markov decision process(CMDP)into an unconstrained optimization problem,which guarantees that the constraints are strictly satisfied without determining the penalty coefficients.Furthermore,to enhance the scalability of the constrained soft actor-critic(CSAC)-based RVPP scheduling approach,a fully distributed scheduling architecture was designed to enable plug-and-play in the residential distributed energy resources(RDER).Case studies performed on the constructed RVPP scenario validated the performance of the proposed methodology in enhancing the responsiveness of the RDER to power tariffs,balancing the supply and demand of the power grid,and ensuring customer comfort.

Actor-critic框架下的二次指派问题求解方法

二次指派问题(QAP)属于NP-hard组合优化问题,在现实生活中有着广泛应用。目前相对成熟的启发式算法通常以问题为导向来设计定制化算法,缺乏迁移泛化能力。为提供一个统一的QAP求解策略,将QAP问题的流量矩阵及距离矩阵抽象成两个无向完全图并构造相应的关联图,从而将设施和地点的指派任务转化为关联图上的节点选择任务,基于actor-critic框架,提出一种全新的求解算法ACQAP。首先,利用多头注意力机制构造策略网络,处理来自图卷积神经网络的节点表征向量;然后,通过actor-critic算法预测每个节点被作为最优节点输出的概率;最后,依据该概率在可行时间内输出满足目标奖励函数的动作决策序列。该算法摆脱人工设计,且适用于不同规模的输入,更加灵活可靠。实验结果表明,在QAPLIB实例上,本算法在精度媲美传统启发式算法的前提下,迁移泛化能力更强;同时相对于NGM等基于学习的算法,求解的指派费用与最优解之间的偏差最小,且在大部分实例中,偏差均小于20%。

基于Actor-Critic自适应PID的钢筋套丝头跟踪检测控制系统研究

为适应流水线节奏,不影响生产进程,从而更好地实现钢筋套丝头质量检测和尺寸测量的自动化与智能化,设计了基于同步带直线导轨的钢筋套丝头检测跟踪系统,并提出了一种基于Actor-Critic的自适应PID控制方法,用强化学习的方法根据环境反馈自动调节PID控制器的比例、积分、微分参数。对该方法和其他PID控制方法的响应性能指标进行实验和分析,实验结果表明该方法能够实现高精度、快速响应的跟踪拍摄,保证高精度的套丝头质量检测。

基于改进Actor-Critic算法的多传感器交叉提示技术

针对在减少战场资源浪费、平衡战场效费比的同时提高目标探测概率,保证目标的可持续跟踪,提出利用改进Actor-Critic算法的多传感器交叉提示技术进行目标探测。首先,综合传感器探测、能耗、时效等因素搭建基于“交叉提示”传感器的动态管理评估模型;其次,重点分析利用Actor-Critic交叉提示算法的传感器管理决策规则,并且提出了Actor-Critic算法,以根据任务自身需求组建中央评价网络,加大传感器与外部环境的交互。仿真结果表明,改进的算法可以加速网络收益,实现对目标的持续性探测,加强传感器之间的交叉提示功能,提升调度的智能化水平,具有较大的应用价值。

广义行为正则化离线Actor-Critic

行为正则化Actor-Critic(BRAC)是一种离线强化学习算法,通过将当前策略与行为策略之间的Kullback-Leibler(KL)散度作为策略目标函数的正则化项来缓解分布偏移问题.但是,由于KL散度是一种无界的分布差异度量,在策略差异过大时,策略目标函数中的累积期望回报项将仅对策略改进发挥有限的作用,从而导致最终学到的策略性能较差.针对该问题,将当前策略与行为策略之间的斜对称Jensen-Shannon(JS)散度作为策略目标函数的正则化项,提出了一种广义行为正则化离线Actor-Critic(GOACBR)算法.理论分析表明:由于斜对称JS散度有界,将其作为正则化项有助于降低策略性能差异.进一步,针对行为策略未知导致难以直接计算当前策略和行为策略间斜对称JS散度的问题,设计了一个辅助网络来对其进行间接估计.最后,给出了GOACBR的收敛性理论证明.在D4RL基准数据集上的评估结果表明:相较于BRAC,GOACBR在所有测试任务上获得的平均累积回报总和提升了289.8%.相关代码公布在https://github.com/houge1996/GOAC.

Challenges and Opportunities in Preserving Key Structural Features of 3D-Printed Metal/Covalent Organic Framework

Metal-organic framework(MOF)and covalent organic framework(COF)are a huge group of advanced porous materials exhibiting attractive and tunable microstructural features,such as large surface area,tunable pore size,and functional surfaces,which have significant values in various application areas.The emerging 3D printing technology further provides MOF and COFs(M/COFs)with higher designability of their macrostructure and demonstrates large achievements in their performance by shaping them into advanced 3D monoliths.However,the currently available 3D printing M/COFs strategy faces a major challenge of severe destruction of M/COFs’microstructural features,both during and after 3D printing.It is envisioned that preserving the microstructure of M/COFs in the 3D-printed monolith will bring a great improvement to the related applications.In this overview,the 3D-printed M/COFs are categorized into M/COF-mixed monoliths and M/COF-covered monoliths.Their differences in the properties,applications,and current research states are discussed.The up-to-date advancements in paste/scaffold composition and printing/covering methods to preserve the superior M/COF microstructure during 3D printing are further discussed for the two types of 3D-printed M/COF.Throughout the analysis of the current states of 3D-printed M/COFs,the expected future research direction to achieve a highly preserved microstructure in the 3D monolith is proposed.

Parallel Light Fields: A Perspective and A Framework

Dear Editor,Light fields give relatively complete description of scenes from perspective of angles and positions of rays. At present time, most of the computer vision algorithms take 2D images as input which are simplified expression of light fields with depth information discarded. In theory, computer vision tasks may achieve better performance as long as complete light fields are acquired.

All‑Covalent Organic Framework Nanofilms Assembled Lithium‑Ion Capacitor to Solve the Imbalanced Charge Storage Kinetics

Free-standing covalent organic framework(COFs)nanofilms exhibit a remarkable ability to rapidly intercalate/de-intercalate Li^(+) in lithium-ion batteries,while simultaneously exposing affluent active sites in supercapacitors.The development of these nanofilms offers a promising solution to address the persistent challenge of imbalanced charge storage kinetics between battery-type anode and capacitor-type cathode in lithium-ion capacitors(LICs).Herein,for the first time,custom-made COFBTMB-TP and COFTAPB-BPY nanofilms are synthesized as the anode and cathode,respectively,for an all-COF nanofilm-structured LIC.The COFBTMB-TP nanofilm with strong electronegative–CF3 groups enables tuning the partial electron cloud density for Li^(+) migration to ensure the rapid anode kinetic process.The thickness-regulated cathodic COFTAPB-BPY nanofilm can fit the anodic COF nanofilm in the capacity.Due to the aligned 1D channel,2D aromatic skeleton and accessible active sites of COF nanofilms,the whole COFTAPB-BPY//COFBTMB-TP LIC demonstrates a high energy density of 318 mWh cm^(−3) at a high-power density of 6 W cm^(−3),excellent rate capability,good cycle stability with the capacity retention rate of 77%after 5000-cycle.The COFTAPB-BPY//COFBTMB-TP LIC represents a new benchmark for currently reported film-type LICs and even film-type supercapacitors.After being comprehensively explored via ex situ XPS,7Li solid-state NMR analyses,and DFT calculation,it is found that the COFBTMB-TP nanofilm facilitates the reversible conversion of semi-ionic to ionic C–F bonds during lithium storage.COFBTMB-TP exhibits a strong interaction with Li^(+) due to the C–F,C=O,and C–N bonds,facilitating Li^(+) desolation and absorption from the electrolyte.This work addresses the challenge of imbalanced charge storage kinetics and capacity between the anode and cathode and also pave the way for future miniaturized and wearable LIC devices.

Accelerating Oxygen Electrocatalysis Kinetics on Metal-Organic Frameworks via Bond Length Optimization

Metal-organic frameworks(MOFs)have been developed as an ideal platform for exploration of the relationship between intrinsic structure and catalytic activity,but the limited catalytic activity and stability has hampered their practical use in water splitting.Herein,we develop a bond length adjustment strategy for optimizing naphthalene-based MOFs that synthesized by acid etching Co-naphthalenedicarboxylic acid-based MOFs(donated as AE-CoNDA)to serve as efficient catalyst for water splitting.AE-CoNDA exhibits a low overpotential of 260 mV to reach 10 mA cm^(−2)and a small Tafel slope of 62 mV dec^(−1)with excellent stability over 100 h.After integrated AE-CoNDA onto BiVO_(4),photocurrent density of 4.3 mA cm^(−2)is achieved at 1.23 V.Experimental investigations demonstrate that the stretched Co-O bond length was found to optimize the orbitals hybridization of Co 3d and O 2p,which accounts for the fast kinetics and high activity.Theoretical calculations reveal that the stretched Co-O bond length strengthens the adsorption of oxygen-contained intermediates at the Co active sites for highly efficient water splitting.

Managing the surge:A comprehensive review of the entire disposal framework for retired lithium-ion batteries from electric vehicles

Anticipating the imminent surge of retired lithium-ion batteries(R-LIBs)from electric vehicles,the need for safe,cost-effective and environmentally friendly disposal technologies has escalated.This paper seeks to offer a comprehensive overview of the entire disposal framework for R-LIBs,encompassing a broad spectrum of activities,including screening,repurposing and recycling.Firstly,we delve deeply into a thorough examination of current screening technologies,shifting the focus from a mere enumeration of screening methods to the exploration of the strategies for enhancing screening efficiency.Secondly,we outline battery repurposing with associated key factors,summarizing stationary applications and sizing methods for R-LIBs in their second life.A particular light is shed on available reconditioning solutions,demonstrating their great potential in facilitating battery safety and lifetime in repurposing scenarios and identifying their techno-economic issues.In the realm of battery recycling,we present an extensive survey of pre-treatment options and subsequent material recovery technologies.Particularly,we introduce several global leading recyclers to illustrate their industrial processes and technical intricacies.Furthermore,relevant challenges and evolving trends are investigated in pursuit of a sustainable end-of-life management and disposal framework.We hope that this study can serve as a valuable resource for researchers,industry professionals and policymakers in this field,ultimately facilitating the adoption of proper disposal practices.

Covalent Organic Framework with 3D Ordered Channel and Multi-Functional Groups Endows Zn Anode with Superior Stability

Achieving a highly robust zinc(Zn)metal anode is extremely important for improving the performance of aqueous Zn-ion batteries(AZIBs)for advancing“carbon neutrality”society,which is hampered by the uncontrollable growth of Zn dendrite and severe side reactions including hydrogen evolution reaction,corrosion,and passivation,etc.Herein,an interlayer containing fluorinated zincophilic covalent organic framework with sulfonic acid groups(COF-S-F)is developed on Zn metal(Zn@COF-S-F)as the artificial solid electrolyte interface(SEI).Sulfonic acid group(-SO_(3)H)in COF-S-F can effectively ameliorate the desolvation process of hydrated Zn ions,and the three-dimensional channel with fluoride group(-F)can provide interconnected channels for the favorable transport of Zn ions with ion-confinement effects,endowing Zn@COF-S-F with dendrite-free morphology and suppressed side reactions.Consequently,Zn@COF-S-F symmetric cell can stably cycle for 1,000 h with low average hysteresis voltage(50.5 m V)at the current density of 1.5 m A cm^(-2).Zn@COF-S-F|Mn O_(2)cell delivers the discharge specific capacity of 206.8 m Ah g^(-1)at the current density of 1.2 A g^(-1)after 800 cycles with high-capacity retention(87.9%).Enlightening,building artificial SEI on metallic Zn surface with targeted design has been proved as the effective strategy to foster the practical application of high-performance AZIBs.

Systematic Security Guideline Framework through Intelligently Automated Vulnerability Analysis

This research aims to propose a practical framework designed for the automatic analysis of a product’s comprehensive functionality and security vulnerabilities,generating applicable guidelines based on real-world software.The existing analysis of software security vulnerabilities often focuses on specific features or modules.This partial and arbitrary analysis of the security vulnerabilities makes it challenging to comprehend the overall security vulnerabilities of the software.The key novelty lies in overcoming the constraints of partial approaches.The proposed framework utilizes data from various sources to create a comprehensive functionality profile,facilitating the derivation of real-world security guidelines.Security guidelines are dynamically generated by associating functional security vulnerabilities with the latest Common Vulnerabilities and Exposure(CVE)and Common Vulnerability Scoring System(CVSS)scores,resulting in automated guidelines tailored to each product.These guidelines are not only practical but also applicable in real-world software,allowing for prioritized security responses.The proposed framework is applied to virtual private network(VPN)software,wherein a validated Level 2 data flow diagram is generated using the Spoofing,Tampering,Repudiation,Information Disclosure,Denial of Service,and Elevation of privilege(STRIDE)technique with references to various papers and examples from related software.The analysis resulted in the identification of a total of 121 vulnerabilities.The successful implementation and validation demonstrate the framework’s efficacy in generating customized guidelines for entire systems,subsystems,and selected modules.

Metal–Organic Framework‑Based Photodetectors

The unique and interesting physical and chemical properties of metal–organic framework(MOF)materials have recently attracted extensive attention in a new generation of photoelectric applications.In this review,we summarized and discussed the research progress on MOF-based photodetectors.The methods of preparing MOF-based photodetectors and various types of MOF single crystals and thin film as well as MOF composites are introduced in details.Additionally,the photodetectors applications for X-ray,ultraviolet and infrared light,biological detectors,and circularly polarized light photodetectors are discussed.Furthermore,summaries and challenges are provided for this important research field.

Metal-organic frameworks and their composites for advanced lithium-ion batteries:Synthesis,progress and prospects

Metal-organic frameworks(MOFs)are among the most promising materials for lithium-ion batteries(LIBs)owing to their high surface area,periodic porosity,adjustable pore size,and controllable chemical composition.For instance,their unique porous structures promote electrolyte penetration,ions transport,and make them ideal for battery separators.Regulating the chemical composition of MOF can introduce more active sites for electrochemical reactions.Therefore,MOFs and their related composites have been extensively and thoroughly explored for LIBs.However,the reported reviews solely include the applications of MOFs in the electrode materials of LIBs and rarely involve other aspects.A systematic review of the application of MOFs in LIBs is essential for understanding the mechanism of MOFs and better designing related MOFs battery materials.This review systematically evaluates the latest developments in pristine MOFs and MOF composites for LIB applications,including MOFs as the main materials(anode,cathode,separators,and electrolytes)to auxiliary materials(coating layers and additives for electrodes).Furthermore,the synthesis,modification methods,challenges,and prospects for the application of MOFs in LIBs are discussed.

Advances of Electrochemical and Electrochemiluminescent Sensors Based on Covalent Organic Frameworks

Covalent organic frameworks(COFs),a rapidly developing category of crystalline conjugated organic polymers,possess highly ordered structures,large specific surface areas,stable chemical properties,and tunable pore microenvironments.Since the first report of boroxine/boronate ester-linked COFs in 2005,COFs have rapidly gained popularity,showing important application prospects in various fields,such as sensing,catalysis,separation,and energy storage.Among them,COFs-based electrochemical(EC)sensors with upgraded analytical performance are arousing extensive interest.In this review,therefore,we summarize the basic properties and the general synthesis methods of COFs used in the field of electroanalytical chemistry,with special emphasis on their usages in the fabrication of chemical sensors,ions sensors,immunosensors,and aptasensors.Notably,the emerged COFs in the electrochemiluminescence(ECL)realm are thoroughly covered along with their preliminary applications.Additionally,final conclusions on state-of-the-art COFs are provided in terms of EC and ECL sensors,as well as challenges and prospects for extending and improving the research and applications of COFs in electroanalytical chemistry.

Hollow Metal-Organic Framework/MXene/Nanocellulose Composite Films for Giga/Terahertz Electromagnetic Shielding and Photothermal Conversion

With the continuous advancement of communication technology,the escalating demand for electromagnetic shielding interference(EMI)materials with multifunctional and wideband EMI performance has become urgent.Controlling the electrical and magnetic components and designing the EMI material structure have attracted extensive interest,but remain a huge challenge.Herein,we reported the alternating electromagnetic structure composite films composed of hollow metal-organic frameworks/layered MXene/nanocellulose(HMN)by alternating vacuum-assisted filtration process.The HMN composite films exhibit excellent EMI shielding effectiveness performance in the GHz frequency(66.8 dB at Kaband)and THz frequency(114.6 dB at 0.1-4.0 THz).Besides,the HMN composite films also exhibit a high reflection loss of 39.7 dB at 0.7 THz with an effective absorption bandwidth up to 2.1 THz.Moreover,HMN composite films show remarkable photothermal conversion performance,which can reach 104.6℃under 2.0 Sun and 235.4℃under 0.8 W cm^(−2),respectively.The unique micro-and macrostructural design structures will absorb more incident electromagnetic waves via interfacial polarization/multiple scattering and produce more heat energy via the local surface plasmon resonance effect.These features make the HMN composite film a promising candidate for advanced EMI devices for future 6G communication and the protection of electronic equipment in cold environments.