Path Planning and Tracking Control for Parking via Soft Actor-Critic Under Non-Ideal Scenarios

Parking in a small parking lot within limited space poses a difficult task. It often leads to deviations between the final parking posture and the target posture. These deviations can lead to partial occupancy of adjacent parking lots, which poses a safety threat to vehicles parked in these parking lots. However, previous studies have not addressed this issue. In this paper, we aim to evaluate the impact of parking deviation of existing vehicles next to the target parking lot(PDEVNTPL) on the automatic ego vehicle(AEV) parking, in terms of safety, comfort, accuracy, and efficiency of parking. A segmented parking training framework(SPTF) based on soft actor-critic(SAC) is proposed to improve parking performance. In the proposed method, the SAC algorithm incorporates strategy entropy into the objective function, to enable the AEV to learn parking strategies based on a more comprehensive understanding of the environment. Additionally, the SPTF simplifies complex parking tasks to maintain the high performance of deep reinforcement learning(DRL). The experimental results reveal that the PDEVNTPL has a detrimental influence on the AEV parking in terms of safety, accuracy, and comfort, leading to reductions of more than 27%, 54%, and 26%respectively. However, the SAC-based SPTF effectively mitigates this impact, resulting in a considerable increase in the parking success rate from 71% to 93%. Furthermore, the heading angle deviation is significantly reduced from 2.25 degrees to 0.43degrees.

融合Dead-ends和离线监督Actor-Critic的动态治疗策略生成模型

强化学习对数学模型依赖性低,利用经验便于架构和优化模型,非常适合用于动态治疗策略学习。但现有研究仍存在以下问题:1)学习策略最优性的同时未考虑风险,导致学到的策略存在一定的风险;2)忽略了分布偏移问题,导致学到的策略与医生策略完全不同;3)忽略患者的历史观测数据和治疗史,从而不能很好地得到患者状态,进而导致不能学到最优策略。基于此,提出了融合Dead-ends和离线监督Actor-Critic的动态治疗策略生成模型DOSAC-DTR。首先,考虑学到的策略所推荐的治疗行动的风险性,在Actor-Critic框架中融入Dead-ends概念;其次,为缓解分布偏移问题,在Actor-Critic框架中融入医生监督,在最大化预期回报的同时,最小化所学策略与医生策略之间的差距;最后,为了得到包含患者关键历史信息的状态表示,使用基于LSTM的编码器解码器模型对患者的历史观测数据和治疗史进行建模。实验结果表明,DOSAC-DTR相比基线方法有更好的性能,可以得到更低的估计死亡率以及更高的Jaccard系数。

Perception Enhanced Deep Deterministic Policy Gradient for Autonomous Driving in Complex Scenarios

Autonomous driving has witnessed rapid advancement;however,ensuring safe and efficient driving in intricate scenarios remains a critical challenge.In particular,traffic roundabouts bring a set of challenges to autonomous driving due to the unpredictable entry and exit of vehicles,susceptibility to traffic flow bottlenecks,and imperfect data in perceiving environmental information,rendering them a vital issue in the practical application of autonomous driving.To address the traffic challenges,this work focused on complex roundabouts with multi-lane and proposed a Perception EnhancedDeepDeterministic Policy Gradient(PE-DDPG)for AutonomousDriving in the Roundabouts.Specifically,themodel incorporates an enhanced variational autoencoder featuring an integrated spatial attention mechanism alongside the Deep Deterministic Policy Gradient framework,enhancing the vehicle’s capability to comprehend complex roundabout environments and make decisions.Furthermore,the PE-DDPG model combines a dynamic path optimization strategy for roundabout scenarios,effectively mitigating traffic bottlenecks and augmenting throughput efficiency.Extensive experiments were conducted with the collaborative simulation platform of CARLA and SUMO,and the experimental results show that the proposed PE-DDPG outperforms the baseline methods in terms of the convergence capacity of the training process,the smoothness of driving and the traffic efficiency with diverse traffic flow patterns and penetration rates of autonomous vehicles(AVs).Generally,the proposed PE-DDPGmodel could be employed for autonomous driving in complex scenarios with imperfect data.

无人机辅助物联网中基于Safe Actor-Critic的信息年龄最小化研究

无人机作为一种新的通信设备,有望在物联网数据采集、监控等业务中发挥关键作用。为保证所采集数据的时效性,利用信息年龄来衡量无人机从物联网设备接收到的数据新鲜度。通过联合优化无人机轨迹和无人机与物联网设备的关联策略以最小化信息年龄加权和,并保证无人机累积飞行能量消耗满足预算要求。由于上述问题同时受短期和长期约束条件的限制,将问题建模为受约束的马尔可夫决策过程(CMDP),并利用Safe Actor-Critic来求解。仿真结果表明,所提算法在最小化信息年龄的同时,能有效保证能量预算。

基于不确定性估计的离线确定型Actor-Critic

Actor-Critic是一种强化学习方法,通过与环境在线试错交互收集样本来学习策略,是求解序贯感知决策问题的有效手段.但是,这种在线交互的主动学习范式在一些复杂真实环境中收集样本时会带来成本和安全问题离线强化学习作为一种基于数据驱动的强化学习范式,强调从静态样本数据集中学习策略,与环境无探索交互,为机器人、自动驾驶、健康护理等真实世界部署应用提供了可行的解决方案,是近年来的研究热点.目前,离线强化学习方法存在学习策略和行为策略之间的分布偏移挑战,针对这个挑战,通常采用策略约束或值函数正则化来限制访问数据集分布之外(Out-Of-Distribution,OOD)的动作,从而导致学习性能过于保守,阻碍了值函数网络的泛化和学习策略的性能提升.为此,本文利用不确定性估计和OOD采样来平衡值函数学习的泛化性和保守性,提出一种基于不确定性估计的离线确定型Actor-Critic方法(Offline Deterministic Actor-Critic based on UncertaintyEstimation,ODACUE).首先,针对确定型策略,给出一种Q值函数的不确定性估计算子定义,理论证明了该算子学到的Q值函数是最优Q值函数的一种悲观估计.然后,将不确定性估计算子应用于确定型Actor-Critic框架中,通过对不确定性估计算子进行凸组合构造Critic学习的目标函数.最后,D4RL基准数据集任务上的实验结果表明:相较于对比算法,ODACUE在11个不同质量等级数据集任务中的总体性能提升最低达9.56%,最高达64.92%.此外,参数分析和消融实验进一步验证了ODACUE的稳定性和泛化能力.

GRU-integrated constrained soft actor-critic learning enabled fully distributed scheduling strategy for residential virtual power plant

In this study,a novel residential virtual power plant(RVPP)scheduling method that leverages a gate recurrent unit(GRU)-integrated deep reinforcement learning(DRL)algorithm is proposed.In the proposed scheme,the GRU-integrated DRL algorithm guides the RVPP to participate effectively in both the day-ahead and real-time markets,lowering the electricity purchase costs and consumption risks for end-users.The Lagrangian relaxation technique is introduced to transform the constrained Markov decision process(CMDP)into an unconstrained optimization problem,which guarantees that the constraints are strictly satisfied without determining the penalty coefficients.Furthermore,to enhance the scalability of the constrained soft actor-critic(CSAC)-based RVPP scheduling approach,a fully distributed scheduling architecture was designed to enable plug-and-play in the residential distributed energy resources(RDER).Case studies performed on the constructed RVPP scenario validated the performance of the proposed methodology in enhancing the responsiveness of the RDER to power tariffs,balancing the supply and demand of the power grid,and ensuring customer comfort.

The deterministic condition for the ground reaction force acting point on the combined knee valgus and tibial internal rotation moments in early phase of cutting maneuvers in female athletes

Background:Combined knee valgus and tibial internal rotation(VL+IR)moments have been shown to stress the anterior cruciate ligament(ACL)in several in vitro cadaveric studies.To utilize this knowledge for non-contact ACL injury prevention in sports,it is necessary to elucidate how the ground reaction force(GRF)acting point(center of pressure(CoP))in the stance foot produces combined knee VL+IR moments in risky maneuvers,such as cuttings.However,the effects of the GRF acting point on the development of the combined knee VL+IR moment in cutting are still unknown.Methods:We first established the deterministic mechanical condition that the CoP position relative to the tibial rotational axis differentiates the GRF vector’s directional probability for developing the combined knee VL+IR moment,and theoretically predicted that when the CoP is posterior to the tibial rotational axis,the GRF vector is more likely to produce the combined knee VL+IR moment than when the CoP is anterior to the tibial rotational axis.Then,we tested a stochastic aspect of our theory in a lab-controlled in vivo experiment.Fourteen females performed 60˚cutting under forefoot/rearfoot strike conditions(10 trials each).The positions of lower limb markers and GRF data were measured,and the knee moment due to GRF vector was calculated.The trials were divided into anterior-and posterior-CoP groups depending on the CoP position relative to the tibial rotational axis at each 10 ms interval from 0 to 100 ms after foot strike,and the occurrence rate of the combined knee VL+IR moment was compared between trial groups.Results:The posterior-CoP group showed significantly higher occurrence rates of the combined knee VL+IR moment(maximum of 82.8%)at every time point than those of the anterior-CoP trials,as theoretically predicted by the deterministic mechanical condition.Conclusion:The rearfoot strikes inducing the posterior CoP should be avoided to reduce the risk of non-contact ACL injury associated with the combined knee VL+IR stress.

Actor-critic框架下的二次指派问题求解方法

二次指派问题(QAP)属于NP-hard组合优化问题,在现实生活中有着广泛应用。目前相对成熟的启发式算法通常以问题为导向来设计定制化算法,缺乏迁移泛化能力。为提供一个统一的QAP求解策略,将QAP问题的流量矩阵及距离矩阵抽象成两个无向完全图并构造相应的关联图,从而将设施和地点的指派任务转化为关联图上的节点选择任务,基于actor-critic框架,提出一种全新的求解算法ACQAP。首先,利用多头注意力机制构造策略网络,处理来自图卷积神经网络的节点表征向量;然后,通过actor-critic算法预测每个节点被作为最优节点输出的概率;最后,依据该概率在可行时间内输出满足目标奖励函数的动作决策序列。该算法摆脱人工设计,且适用于不同规模的输入,更加灵活可靠。实验结果表明,在QAPLIB实例上,本算法在精度媲美传统启发式算法的前提下,迁移泛化能力更强;同时相对于NGM等基于学习的算法,求解的指派费用与最优解之间的偏差最小,且在大部分实例中,偏差均小于20%。

基于Actor-Critic自适应PID的钢筋套丝头跟踪检测控制系统研究

为适应流水线节奏,不影响生产进程,从而更好地实现钢筋套丝头质量检测和尺寸测量的自动化与智能化,设计了基于同步带直线导轨的钢筋套丝头检测跟踪系统,并提出了一种基于Actor-Critic的自适应PID控制方法,用强化学习的方法根据环境反馈自动调节PID控制器的比例、积分、微分参数。对该方法和其他PID控制方法的响应性能指标进行实验和分析,实验结果表明该方法能够实现高精度、快速响应的跟踪拍摄,保证高精度的套丝头质量检测。

基于改进Actor-Critic算法的多传感器交叉提示技术

针对在减少战场资源浪费、平衡战场效费比的同时提高目标探测概率,保证目标的可持续跟踪,提出利用改进Actor-Critic算法的多传感器交叉提示技术进行目标探测。首先,综合传感器探测、能耗、时效等因素搭建基于“交叉提示”传感器的动态管理评估模型;其次,重点分析利用Actor-Critic交叉提示算法的传感器管理决策规则,并且提出了Actor-Critic算法,以根据任务自身需求组建中央评价网络,加大传感器与外部环境的交互。仿真结果表明,改进的算法可以加速网络收益,实现对目标的持续性探测,加强传感器之间的交叉提示功能,提升调度的智能化水平,具有较大的应用价值。

广义行为正则化离线Actor-Critic

行为正则化Actor-Critic(BRAC)是一种离线强化学习算法,通过将当前策略与行为策略之间的Kullback-Leibler(KL)散度作为策略目标函数的正则化项来缓解分布偏移问题.但是,由于KL散度是一种无界的分布差异度量,在策略差异过大时,策略目标函数中的累积期望回报项将仅对策略改进发挥有限的作用,从而导致最终学到的策略性能较差.针对该问题,将当前策略与行为策略之间的斜对称Jensen-Shannon(JS)散度作为策略目标函数的正则化项,提出了一种广义行为正则化离线Actor-Critic(GOACBR)算法.理论分析表明:由于斜对称JS散度有界,将其作为正则化项有助于降低策略性能差异.进一步,针对行为策略未知导致难以直接计算当前策略和行为策略间斜对称JS散度的问题,设计了一个辅助网络来对其进行间接估计.最后,给出了GOACBR的收敛性理论证明.在D4RL基准数据集上的评估结果表明:相较于BRAC,GOACBR在所有测试任务上获得的平均累积回报总和提升了289.8%.相关代码公布在https://github.com/houge1996/GOAC.

A Unique Discrete Wavelet&Deterministic Walk-Based Glaucoma Classification Approach Using Image-Specific Enhanced Retinal Images

Glaucoma is a group of ocular atrophy diseases that cause progressive vision loss by affecting the optic nerve.Because of its asymptomatic nature,glaucoma has become the leading cause of human blindness worldwide.In this paper,a novel computer-aided diagnosis(CAD)approach for glaucomatous retinal image classification has been introduced.It extracts graph-based texture features from structurally improved fundus images using discrete wavelet-transformation(DWT)and deterministic tree-walk(DTW)procedures.Retinal images are considered from both public repositories and eye hospitals.Images are enhanced with image-specific luminance and gradient transitions for both contrast and texture improvement.The enhanced images are mapped into undirected graphs using DTW trajectories formed by the image’s wavelet coefficients.Graph-based features are extracted fromthese graphs to capture image texture patterns.Machine learning(ML)classifiers use these features to label retinal images.This approach has attained an accuracy range of 93.5%to 100%,82.1%to 99.3%,95.4%to 100%,83.3%to 96.6%,77.7%to 88.8%,and 91.4%to 100%on the ACRIMA,ORIGA,RIM-ONE,Drishti,HRF,and HOSPITAL datasets,respectively.The major strength of this approach is texture pattern identification using various topological graphs.It has achieved optimal performance with SVM and RF classifiers using biorthogonal DWT combinations on both public and patients’fundus datasets.The classification performance of the DWT-DTW approach is on par with the contemporary state-of-the-art methods,which can be helpful for ophthalmologists in glaucoma screening.

Subspace Identification for Closed-Loop Systems With Unknown Deterministic Disturbances

This paper presents a subspace identification method for closed-loop systems with unknown deterministic disturbances.To deal with the unknown deterministic disturbances,two strategies are implemented to construct the row space that can be used to approximately represent the unknown deterministic disturbances using the trigonometric functions or Bernstein polynomials depending on whether the disturbance frequencies are known.For closed-loop identification,CCF-N4SID is extended to the case with unknown deterministic disturbances using the oblique projection.In addition,a proper Bernstein polynomial order can be determined using the Akaike information criterion(AIC)or the Bayesian information criterion(BIC).Numerical simulation results demonstrate the effectiveness of the proposed identification method for both periodic and aperiodic deterministic disturbances.

Quantization of the Kinetic Energy of Deterministic Chaos

In previous works, the theoretical and experimental deterministic scalar kinematic structures, the theoretical and experimental deterministic vector kinematic structures, the theoretical and experimental deterministic scalar dynamic structures, and the theoretical and experimental deterministic vector dynamic structures have been developed to compute the exact solution for deterministic chaos of the exponential pulsons and oscillons that is governed by the nonstationary three-dimensional Navier-Stokes equations. To explore properties of the kinetic energy, rectangular, diagonal, and triangular summations of a matrix of the kinetic energy and general terms of various sums have been used in the current paper to develop quantization of the kinetic energy of deterministic chaos. Nested structures of a cumulative energy pulson, an energy pulson of propagation, an internal energy oscillon, a diagonal energy oscillon, and an external energy oscillon have been established. In turn, the energy pulsons and oscillons include group pulsons of propagation, internal group oscillons, diagonal group oscillons, and external group oscillons. Sequentially, the group pulsons and oscillons contain wave pulsons of propagation, internal wave oscillons, diagonal wave oscillons, and external wave oscillons. Consecutively, the wave pulsons and oscillons are composed of elementary pulsons of propagation, internal elementary oscillons, diagonal elementary oscillons, and external elementary oscillons. Topology, periodicity, and integral properties of the exponential pulsons and oscillons have been studied using the novel method of the inhomogeneous Fourier expansions via eigenfunctions in coordinates and time. Symbolic computations of the exact expansions have been performed using the experimental and theoretical programming in Maple. Results of the symbolic computations have been justified by probe visualizations.

Actor-Critic框架下的数据驱动异步电机离线参数辨识方法

电动汽车用电机的参数辨识可以使电机在任意转速下尽可能输出更高的转矩及效率,是优化电机输出性能的重要手段。传统的基于模型驱动的参数辨识方法的缺点为易受模型误差的影响、抗干扰能力差以及无法实现全转速范围内的转矩最优。鉴于上述缺点,该文研究了一种完全基于实际数据的电动汽车用异步电机离线参数辨识方法,对电机的转子电阻和励磁电感在任意转速下进行了优化,从而使电机能够在特定转速和特定电流下输出最优转矩。为达到电机在特定转速和电流下输出转矩最优的目的,研究了一种基于Actor-Critic框架的电动汽车用异步电机离线参数辨识方法,确定了框架中的观测、奖励和动作的设计。实验证明相对于传统参数辨识方法,该文方法具有更高的精确性和鲁棒性,同时确保了电动汽车用异步电机在任意转速下的输出转矩最优。

PARETO FRONT CAPTURE USING DETERMINISTIC OPTIMIZATION METHODS IN MULTI-CRITERION AERODYNAMIC DESIGN

Deterministic optimization methods are combined with the Pareto front concept to solve multi-criterion design problems. The algorithm and the numerical implementation are applied to aerodynamic designs. Evolutionary algorithms （EAs） and the Pareto front concept are used to solve practical design problems in industry for its robustness in capturing convex, concave, discrete or discontinuous Pareto fronts of multi-objective optimization problems. However, the process is time-consuming. Therefore, deterministic optimization methods are introduced to capture the Pareto front, and the types of the captured Pareto front are explained. Numerical experiments show that the deterministic optimization method is a good alternative to EAs for capturing any convex and some concave Pareto fronts in multi-criterion aerodynamic optimization problems due to its efficiency.

基于Tile Coding编码和模型学习的Actor-Critic算法

Actor-Critic是一类具有较好性能及收敛保证的强化学习方法,然而,Agent在学习和改进策略的过程中并没有对环境的动态性进行学习,导致Actor-Critic方法的性能受到一定限制。此外,Actor-Critic方法中需要近似地表示策略以及值函数,其中状态和动作的编码方法以及参数对Actor-Critic方法有重要的影响。Tile Coding编码具有简单易用、计算时间复杂度较低等优点,因此,将Tile Coding编码与基于模型的Actor-Critic方法结合,并将所得算法应用于强化学习仿真实验。实验结果表明,所得算法具有较好的性能。

基于批量递归最小二乘的自然Actor-Critic算法

为了减轻Actor-Critic结构中智能体用最小二乘法估计自然梯度时的在线运算负担,提高运算实时性,提出新的学习算法:NAC-BRLS.该算法在Critic中利用批量递归最小二乘法估计自然梯度,根据估计得到的梯度乐观地更新策略.批量递归最小二乘法的引入使得智能体能根据自身运算能力自由调整各批次运算的数据量,即每次策略估计时使用的数据量,在全乐观和部分乐观之间进行权衡,大大提高了NAC-LSTD算法的灵活性.山地车仿真实验表明,与NAC-LSTD算法相比,NAC-BRLS算法在保证一定收敛性能的前提下,能够明显降低智能体的单步平均运算负担.

Actor-Critic框架下的多智能体决策方法及其在兵棋上的应用

将人工智能应用于兵棋推演的智能战术兵棋正逐年发展,基于Actor-Critic框架的决策方法可以实现智能战术兵棋的战术行动动态决策。但若Critic网络只对单算子进行评价,多算子之间的网络没有协同,本方算子之间各自行动决策会不够智能。针对上述方法的不足,提出了一种基于强化学习并结合规则的多智能体决策方法,以提升兵棋推演的智能水平。侧重采用强化学习对多算子的行动决策进行决策分析,并结合产生式规则对战术决策进行规划。构建基于Actor-Critic框架的多算子分布执行集中训练的行动决策模型,对比每个算子互不交流的封闭式行动决策学习方法,提出的分布执行集中训练方法更具优势且有效。

基于Actor-Critic算法的多无人机协同空战目标重分配方法

目标重分配问题是多无人机协同空战中亟需解决的关键问题之一。考虑到空战中的不确定性、实时性等特点,建立了多无人机协同空战目标重分配问题的数学模型,结合强化学习核心概念,提出了基于Actor-Critic算法的多无人机协同空战目标重分配框架,构建了基于目标重分配的马尔科夫决策过程、Actor网络结构和Critic网络结构。针对强化学习算法中存在的奖励稀疏问题,设计了局部回报和全局汇报相结合的双层回报函数。在基于VR-Forces仿真平台中验证了该方法的有效性。实验结果表明,提出的多无人机协同空战目标重分配方法能够有效地提升空战对抗的胜率。