Path Planning and Tracking Control for Parking via Soft Actor-Critic Under Non-Ideal Scenarios

Parking in a small parking lot within limited space poses a difficult task. It often leads to deviations between the final parking posture and the target posture. These deviations can lead to partial occupancy of adjacent parking lots, which poses a safety threat to vehicles parked in these parking lots. However, previous studies have not addressed this issue. In this paper, we aim to evaluate the impact of parking deviation of existing vehicles next to the target parking lot(PDEVNTPL) on the automatic ego vehicle(AEV) parking, in terms of safety, comfort, accuracy, and efficiency of parking. A segmented parking training framework(SPTF) based on soft actor-critic(SAC) is proposed to improve parking performance. In the proposed method, the SAC algorithm incorporates strategy entropy into the objective function, to enable the AEV to learn parking strategies based on a more comprehensive understanding of the environment. Additionally, the SPTF simplifies complex parking tasks to maintain the high performance of deep reinforcement learning(DRL). The experimental results reveal that the PDEVNTPL has a detrimental influence on the AEV parking in terms of safety, accuracy, and comfort, leading to reductions of more than 27%, 54%, and 26%respectively. However, the SAC-based SPTF effectively mitigates this impact, resulting in a considerable increase in the parking success rate from 71% to 93%. Furthermore, the heading angle deviation is significantly reduced from 2.25 degrees to 0.43degrees.

融合Dead-ends和离线监督Actor-Critic的动态治疗策略生成模型

强化学习对数学模型依赖性低,利用经验便于架构和优化模型,非常适合用于动态治疗策略学习。但现有研究仍存在以下问题:1)学习策略最优性的同时未考虑风险,导致学到的策略存在一定的风险;2)忽略了分布偏移问题,导致学到的策略与医生策略完全不同;3)忽略患者的历史观测数据和治疗史,从而不能很好地得到患者状态,进而导致不能学到最优策略。基于此,提出了融合Dead-ends和离线监督Actor-Critic的动态治疗策略生成模型DOSAC-DTR。首先,考虑学到的策略所推荐的治疗行动的风险性,在Actor-Critic框架中融入Dead-ends概念;其次,为缓解分布偏移问题,在Actor-Critic框架中融入医生监督,在最大化预期回报的同时,最小化所学策略与医生策略之间的差距;最后,为了得到包含患者关键历史信息的状态表示,使用基于LSTM的编码器解码器模型对患者的历史观测数据和治疗史进行建模。实验结果表明,DOSAC-DTR相比基线方法有更好的性能,可以得到更低的估计死亡率以及更高的Jaccard系数。

无人机辅助物联网中基于Safe Actor-Critic的信息年龄最小化研究

无人机作为一种新的通信设备,有望在物联网数据采集、监控等业务中发挥关键作用。为保证所采集数据的时效性,利用信息年龄来衡量无人机从物联网设备接收到的数据新鲜度。通过联合优化无人机轨迹和无人机与物联网设备的关联策略以最小化信息年龄加权和,并保证无人机累积飞行能量消耗满足预算要求。由于上述问题同时受短期和长期约束条件的限制,将问题建模为受约束的马尔可夫决策过程(CMDP),并利用Safe Actor-Critic来求解。仿真结果表明,所提算法在最小化信息年龄的同时,能有效保证能量预算。

基于不确定性估计的离线确定型Actor-Critic

Actor-Critic是一种强化学习方法,通过与环境在线试错交互收集样本来学习策略,是求解序贯感知决策问题的有效手段.但是,这种在线交互的主动学习范式在一些复杂真实环境中收集样本时会带来成本和安全问题离线强化学习作为一种基于数据驱动的强化学习范式,强调从静态样本数据集中学习策略,与环境无探索交互,为机器人、自动驾驶、健康护理等真实世界部署应用提供了可行的解决方案,是近年来的研究热点.目前,离线强化学习方法存在学习策略和行为策略之间的分布偏移挑战,针对这个挑战,通常采用策略约束或值函数正则化来限制访问数据集分布之外(Out-Of-Distribution,OOD)的动作,从而导致学习性能过于保守,阻碍了值函数网络的泛化和学习策略的性能提升.为此,本文利用不确定性估计和OOD采样来平衡值函数学习的泛化性和保守性,提出一种基于不确定性估计的离线确定型Actor-Critic方法(Offline Deterministic Actor-Critic based on UncertaintyEstimation,ODACUE).首先,针对确定型策略,给出一种Q值函数的不确定性估计算子定义,理论证明了该算子学到的Q值函数是最优Q值函数的一种悲观估计.然后,将不确定性估计算子应用于确定型Actor-Critic框架中,通过对不确定性估计算子进行凸组合构造Critic学习的目标函数.最后,D4RL基准数据集任务上的实验结果表明:相较于对比算法,ODACUE在11个不同质量等级数据集任务中的总体性能提升最低达9.56%,最高达64.92%.此外,参数分析和消融实验进一步验证了ODACUE的稳定性和泛化能力.

GRU-integrated constrained soft actor-critic learning enabled fully distributed scheduling strategy for residential virtual power plant

In this study,a novel residential virtual power plant(RVPP)scheduling method that leverages a gate recurrent unit(GRU)-integrated deep reinforcement learning(DRL)algorithm is proposed.In the proposed scheme,the GRU-integrated DRL algorithm guides the RVPP to participate effectively in both the day-ahead and real-time markets,lowering the electricity purchase costs and consumption risks for end-users.The Lagrangian relaxation technique is introduced to transform the constrained Markov decision process(CMDP)into an unconstrained optimization problem,which guarantees that the constraints are strictly satisfied without determining the penalty coefficients.Furthermore,to enhance the scalability of the constrained soft actor-critic(CSAC)-based RVPP scheduling approach,a fully distributed scheduling architecture was designed to enable plug-and-play in the residential distributed energy resources(RDER).Case studies performed on the constructed RVPP scenario validated the performance of the proposed methodology in enhancing the responsiveness of the RDER to power tariffs,balancing the supply and demand of the power grid,and ensuring customer comfort.

Actor-critic框架下的二次指派问题求解方法

二次指派问题(QAP)属于NP-hard组合优化问题,在现实生活中有着广泛应用。目前相对成熟的启发式算法通常以问题为导向来设计定制化算法,缺乏迁移泛化能力。为提供一个统一的QAP求解策略,将QAP问题的流量矩阵及距离矩阵抽象成两个无向完全图并构造相应的关联图,从而将设施和地点的指派任务转化为关联图上的节点选择任务,基于actor-critic框架,提出一种全新的求解算法ACQAP。首先,利用多头注意力机制构造策略网络,处理来自图卷积神经网络的节点表征向量;然后,通过actor-critic算法预测每个节点被作为最优节点输出的概率;最后,依据该概率在可行时间内输出满足目标奖励函数的动作决策序列。该算法摆脱人工设计,且适用于不同规模的输入,更加灵活可靠。实验结果表明,在QAPLIB实例上,本算法在精度媲美传统启发式算法的前提下,迁移泛化能力更强;同时相对于NGM等基于学习的算法,求解的指派费用与最优解之间的偏差最小,且在大部分实例中,偏差均小于20%。

基于Actor-Critic自适应PID的钢筋套丝头跟踪检测控制系统研究

为适应流水线节奏,不影响生产进程,从而更好地实现钢筋套丝头质量检测和尺寸测量的自动化与智能化,设计了基于同步带直线导轨的钢筋套丝头检测跟踪系统,并提出了一种基于Actor-Critic的自适应PID控制方法,用强化学习的方法根据环境反馈自动调节PID控制器的比例、积分、微分参数。对该方法和其他PID控制方法的响应性能指标进行实验和分析,实验结果表明该方法能够实现高精度、快速响应的跟踪拍摄,保证高精度的套丝头质量检测。

基于改进Actor-Critic算法的多传感器交叉提示技术

针对在减少战场资源浪费、平衡战场效费比的同时提高目标探测概率,保证目标的可持续跟踪,提出利用改进Actor-Critic算法的多传感器交叉提示技术进行目标探测。首先,综合传感器探测、能耗、时效等因素搭建基于“交叉提示”传感器的动态管理评估模型;其次,重点分析利用Actor-Critic交叉提示算法的传感器管理决策规则,并且提出了Actor-Critic算法,以根据任务自身需求组建中央评价网络,加大传感器与外部环境的交互。仿真结果表明,改进的算法可以加速网络收益,实现对目标的持续性探测,加强传感器之间的交叉提示功能,提升调度的智能化水平,具有较大的应用价值。

采用STAMP-24Model的多组织事故分析

安全生产事故往往由多组织交互、多因素耦合造成,事故原因涉及多个组织。为预防和遏制多组织生产安全事故的发生,基于系统理论事故建模与过程模型(Systems-Theory Accident Modeling and Process,STAMP)、24Model,构建一种用于多组织事故分析的方法,并以青岛石油爆炸事故为例进行事故原因分析。结果显示:STAMP-24Model可以分组织,分层次且有效、全面、详细地分析涉及多个组织的事故原因,探究多组织之间的交互关系;对事故进行动态演化分析,可得到各组织不安全动作耦合关系与形成的事故失效链及管控失效路径,进而为预防多组织事故提供思路和参考。

基于改进24Model-ISM-SNA建筑工人不安全行为关联路径研究

建筑施工现场环境复杂,为有效控制不安全行为发生,基于行为安全“2-4”模型对360份具有代表性的建筑安全事故调查报告进行分析,提取出22个不安全行为的主要影响因素。利用灰色关联分析方法(GRA)改进的集成ISM-SNA模型,将不安全行为风险因素划分为表层、过渡层与深层,然后对风险因素进行可视化分析、中心度分析及凝聚子群分析,揭示了各致因因素间的关联关系和传导路径。结果表明,建筑工人不安全行为影响因素可划分成7级3阶的多级递阶结构,安全意识、现场监管、外部环境是建筑工人不安全行为的关键影响因素,同时现场监管和隐患排查到位能有效降低不安全行为的发生。

基于24Model-D-ISM的地铁站火灾疏散影响因素研究

为预防地铁站火灾事故,深入了解地铁站火灾人员疏散影响因素间的内在联系与层次结构,基于第6版“2-4”模型(24Model)分析63起地铁站火灾疏散事故,充分考虑各个因素之间的交互作用,提取19个影响地铁站人员疏散的关键因素,建立地铁站火灾人员疏散影响因素指标体系;采用算子客观赋权法(C-OWA)改进决策试验与评价实验法(DEMATEL),确定地铁站火灾人员疏散的重要影响因素;在此基础上,采用解释结构模型(ISM)分析各个因素间的层次结构及相互作用路径,构建地铁站火灾人员疏散影响因素的多级递阶结构模型。研究结果表明:疏散引导、恐慌从众行为、人员拥挤为地铁站火灾人员疏散的关键影响因素;地铁站火灾人员疏散受表层因素、中间层因素、深层因素共同作用的影响,其中,疏散教育与培训、设施维护与检查、疏散预案等因素是根源影响因素,重视根源影响因素的改善有利于从本质上预防和控制事故的发生。

广义行为正则化离线Actor-Critic

行为正则化Actor-Critic(BRAC)是一种离线强化学习算法,通过将当前策略与行为策略之间的Kullback-Leibler(KL)散度作为策略目标函数的正则化项来缓解分布偏移问题.但是,由于KL散度是一种无界的分布差异度量,在策略差异过大时,策略目标函数中的累积期望回报项将仅对策略改进发挥有限的作用,从而导致最终学到的策略性能较差.针对该问题,将当前策略与行为策略之间的斜对称Jensen-Shannon(JS)散度作为策略目标函数的正则化项,提出了一种广义行为正则化离线Actor-Critic(GOACBR)算法.理论分析表明:由于斜对称JS散度有界,将其作为正则化项有助于降低策略性能差异.进一步,针对行为策略未知导致难以直接计算当前策略和行为策略间斜对称JS散度的问题,设计了一个辅助网络来对其进行间接估计.最后,给出了GOACBR的收敛性理论证明.在D4RL基准数据集上的评估结果表明:相较于BRAC,GOACBR在所有测试任务上获得的平均累积回报总和提升了289.8%.相关代码公布在https://github.com/houge1996/GOAC.

Projecting Wintertime Newly Formed Arctic Sea Ice through Weighting CMIP6 Model Performance and Independence

Precipitous Arctic sea-ice decline and the corresponding increase in Arctic open-water areas in summer months give more space for sea-ice growth in the subsequent cold seasons. Compared to the decline of the entire Arctic multiyear sea ice,changes in newly formed sea ice indicate more thermodynamic and dynamic information on Arctic atmosphere–ocean–ice interaction and northern mid–high latitude atmospheric teleconnections. Here, we use a large multimodel ensemble from phase 6 of the Coupled Model Intercomparison Project(CMIP6) to investigate future changes in wintertime newly formed Arctic sea ice. The commonly used model-democracy approach that gives equal weight to each model essentially assumes that all models are independent and equally plausible, which contradicts with the fact that there are large interdependencies in the ensemble and discrepancies in models' performances in reproducing observations. Therefore, instead of using the arithmetic mean of well-performing models or all available models for projections like in previous studies, we employ a newly developed model weighting scheme that weights all models in the ensemble with consideration of their performance and independence to provide more reliable projections. Model democracy leads to evident bias and large intermodel spread in CMIP6 projections of newly formed Arctic sea ice. However, we show that both the bias and the intermodel spread can be effectively reduced by the weighting scheme. Projections from the weighted models indicate that wintertime newly formed Arctic sea ice is likely to increase dramatically until the middle of this century regardless of the emissions scenario.Thereafter, it may decrease(or remain stable) if the Arctic warming crosses a threshold(or is extensively constrained).

Anisotropic time-dependent behaviors of shale under direct shearing and associated empirical creep models

Understanding the anisotropic creep behaviors of shale under direct shearing is a challenging issue.In this context,we conducted shear-creep and steady-creep tests on shale with five bedding orientations (i.e.0°,30°,45°,60°,and 90°),under multiple levels of direct shearing for the first time.The results show that the anisotropic creep of shale exhibits a significant stress-dependent behavior.Under a low shear stress,the creep compliance of shale increases linearly with the logarithm of time at all bedding orientations,and the increase depends on the bedding orientation and creep time.Under high shear stress conditions,the creep compliance of shale is minimal when the bedding orientation is 0°,and the steady-creep rate of shale increases significantly with increasing bedding orientations of 30°,45°,60°,and 90°.The stress-strain values corresponding to the inception of the accelerated creep stage show an increasing and then decreasing trend with the bedding orientation.A semilogarithmic model that could reflect the stress dependence of the steady-creep rate while considering the hardening and damage process is proposed.The model minimizes the deviation of the calculated steady-state creep rate from the observed value and reveals the behavior of the bedding orientation's influence on the steady-creep rate.The applicability of the five classical empirical creep models is quantitatively evaluated.It shows that the logarithmic model can well explain the experimental creep strain and creep rate,and it can accurately predict long-term shear creep deformation.Based on an improved logarithmic model,the variations in creep parameters with shear stress and bedding orientations are discussed.With abovementioned findings,a mathematical method for constructing an anisotropic shear creep model of shale is proposed,which can characterize the nonlinear dependence of the anisotropic shear creep behavior of shale on the bedding orientation.

Geostatistical seismic inversion and 3D modelling of metric flow units,porosity and permeability in Brazilian presalt reservoir

Flow units(FU)rock typing is a common technique for characterizing reservoir flow behavior,producing reliable porosity and permeability estimation even in complex geological settings.However,the lateral extrapolation of FU away from the well into the whole reservoir grid is commonly a difficult task and using the seismic data as constraints is rarely a subject of study.This paper proposes a workflow to generate numerous possible 3D volumes of flow units,porosity and permeability below the seismic resolution limit,respecting the available seismic data at larger scales.The methodology is used in the Mero Field,a Brazilian presalt carbonate reservoir located in the Santos Basin,who presents a complex and heterogenic geological setting with different sedimentological processes and diagenetic history.We generated metric flow units using the conventional core analysis and transposed to the well log data.Then,given a Markov chain Monte Carlo algorithm,the seismic data and the well log statistics,we simulated acoustic impedance,decametric flow units(DFU),metric flow units(MFU),porosity and permeability volumes in the metric scale.The aim is to estimate a minimum amount of MFU able to calculate realistic scenarios porosity and permeability scenarios,without losing the seismic lateral control.In other words,every porosity and permeability volume simulated produces a synthetic seismic that match the real seismic of the area,even in the metric scale.The achieved 3D results represent a high-resolution fluid flow reservoir modelling considering the lateral control of the seismic during the process and can be directly incorporated in the dynamic characterization workflow.

耦合优化蚁群算法与P-Median model的选址模型设计

为节省城建部门对于公共体育设施的投入成本以及提高城市人民生活质量,以运动场所优化选址为例,提出一种新型设施选址模型。该模型主要基于P-Median model(最小化阻抗模型)根据需求点数量从全部候选设施选址中选择设施空间位置,让用户达到离自己最近设施距离成本总和最小的目的,对选址的基本原则和实际情况提出要求,构造目标函数用于优化后蚁群算法求解进行选址工作。优化蚁群算法实现基于Python语言模块,通过改进蚁群原始信息素,提升原有算法的收敛速度,求出目标函数最优解,可以很好地模拟对于运动场所的选址。用二者耦合进行优势互补所设计的选址模型来搜寻研究区蚁群信息素浓度残留最大的栅格像元,从而确定未被已有设施点服务半径覆盖的最佳设施点建立位置。实验结果表明,该新型选址模型相较于最小化阻抗模型与最大化覆盖模型,新增优化设施点使整体服务半径覆盖率分别高出10.42%和6.95%,适合求解较为精确且小规模空间下的选址问题。

A modified stochastic model for LS+AR hybrid method and its application in polar motion short-term prediction

Short-term(up to 30 days)predictions of Earth Rotation Parameters(ERPs)such as Polar Motion(PM:PMX and PMY)play an essential role in real-time applications related to high-precision reference frame conversion.Currently,least squares(LS)+auto-regressive(AR)hybrid method is one of the main techniques of PM prediction.Besides,the weighted LS+AR hybrid method performs well for PM short-term prediction.However,the corresponding covariance information of LS fitting residuals deserves further exploration in the AR model.In this study,we have derived a modified stochastic model for the LS+AR hybrid method,namely the weighted LS+weighted AR hybrid method.By using the PM data products of IERS EOP 14 C04,the numerical results indicate that for PM short-term forecasting,the proposed weighted LS+weighted AR hybrid method shows an advantage over both the LS+AR hybrid method and the weighted LS+AR hybrid method.Compared to the mean absolute errors(MAEs)of PMX/PMY sho rt-term prediction of the LS+AR hybrid method and the weighted LS+AR hybrid method,the weighted LS+weighted AR hybrid method shows average improvements of 6.61%/12.08%and 0.24%/11.65%,respectively.Besides,for the slopes of the linear regression lines fitted to the errors of each method,the growth of the prediction error of the proposed method is slower than that of the other two methods.

Mshpy23:a user-friendly,parameterized model of magnetosheath conditions

Lunar Environment heliospheric X-ray Imager(LEXI)and Solar wind−Magnetosphere−Ionosphere Link Explorer(SMILE)will observe magnetosheath and its boundary motion in soft X-rays for understanding magnetopause reconnection modes under various solar wind conditions after their respective launches in 2024 and 2025.Magnetosheath conditions,namely,plasma density,velocity,and temperature,are key parameters for predicting and analyzing soft X-ray images from the LEXI and SMILE missions.We developed a userfriendly model of magnetosheath that parameterizes number density,velocity,temperature,and magnetic field by utilizing the global Magnetohydrodynamics(MHD)model as well as the pre-existing gas-dynamic and analytic models.Using this parameterized magnetosheath model,scientists can easily reconstruct expected soft X-ray images and utilize them for analysis of observed images of LEXI and SMILE without simulating the complicated global magnetosphere models.First,we created an MHD-based magnetosheath model by running a total of 14 OpenGGCM global MHD simulations under 7 solar wind densities(1,5,10,15,20,25,and 30 cm)and 2 interplanetary magnetic field Bz components(±4 nT),and then parameterizing the results in new magnetosheath conditions.We compared the magnetosheath model result with THEMIS statistical data and it showed good agreement with a weighted Pearson correlation coefficient greater than 0.77,especially for plasma density and plasma velocity.Second,we compiled a suite of magnetosheath models incorporating previous magnetosheath models(gas-dynamic,analytic),and did two case studies to test the performance.The MHD-based model was comparable to or better than the previous models while providing self-consistency among the magnetosheath parameters.Third,we constructed a tool to calculate a soft X-ray image from any given vantage point,which can support the planning and data analysis of the aforementioned LEXI and SMILE missions.A release of the code has been uploaded to a Github repository.

Development and validation of a prediction model for early screening of people at high risk for colorectal cancer

BACKGROUND Colorectal cancer(CRC)is a serious threat worldwide.Although early screening is suggested to be the most effective method to prevent and control CRC,the current situation of early screening for CRC is still not optimistic.In China,the incidence of CRC in the Yangtze River Delta region is increasing dramatically,but few studies have been conducted.Therefore,it is necessary to develop a simple and efficient early screening model for CRC.AIM To develop and validate an early-screening nomogram model to identify individuals at high risk of CRC.METHODS Data of 64448 participants obtained from Ningbo Hospital,China between 2014 and 2017 were retrospectively analyzed.The cohort comprised 64448 individuals,of which,530 were excluded due to missing or incorrect data.Of 63918,7607(11.9%)individuals were considered to be high risk for CRC,and 56311(88.1%)were not.The participants were randomly allocated to a training set(44743)or validation set(19175).The discriminatory ability,predictive accuracy,and clinical utility of the model were evaluated by constructing and analyzing receiver operating characteristic(ROC)curves and calibration curves and by decision curve analysis.Finally,the model was validated internally using a bootstrap resampling technique.RESULTS Seven variables,including demographic,lifestyle,and family history information,were examined.Multifactorial logistic regression analysis revealed that age[odds ratio(OR):1.03,95%confidence interval(CI):1.02-1.03,P<0.001],body mass index(BMI)(OR:1.07,95%CI:1.06-1.08,P<0.001),waist circumference(WC)(OR:1.03,95%CI:1.02-1.03 P<0.001),lifestyle(OR:0.45,95%CI:0.42-0.48,P<0.001),and family history(OR:4.28,95%CI:4.04-4.54,P<0.001)were the most significant predictors of high-risk CRC.Healthy lifestyle was a protective factor,whereas family history was the most significant risk factor.The area under the curve was 0.734(95%CI:0.723-0.745)for the final validation set ROC curve and 0.735(95%CI:0.728-0.742)for the training set ROC curve.The calibration curve demonstrated a high correlation between the CRC high-risk population predicted by the nomogram model and the actual CRC high-risk population.CONCLUSION The early-screening nomogram model for CRC prediction in high-risk populations developed in this study based on age,BMI,WC,lifestyle,and family history exhibited high accuracy.

24Model与LCM原因因素定义对比研究

为探究损失致因模型(LCM)原因因素定义与事故致因“2-4”模型(24Model)存在的异同和优缺点,梳理2个模型各层面原因和结果的定义,对比定义内容及其对事故原因分析等安全实务的指导作用,并以一起瓦斯爆炸事故为例加以实证分析,获得二者分析结果之间的差异。研究结果表明:LCM是首个将管理因素纳入事故致因分析的一维事件序列模型,可明确各层面原因因素的定义和因素间的逻辑关系,但部分定义存在交叉重复的问题,并没有揭示安全工作指导思想等深层次事故致因因素;24Model作为系统性事故致因模型,对各类因素的定义均以组织为主体,描述事件、事故、安全的概念内涵,划分个体安全动作、安全能力和组织安全管理体系的类别并给出含义解析,探究组织安全文化层面的问题并以32个元素体现;2个模型的事故原因分析方法均建立在对各层级原因因素定义的基础上,并适用于模型理论体系本身。