Path Planning and Tracking Control for Parking via Soft Actor-Critic Under Non-Ideal Scenarios

Parking in a small parking lot within limited space poses a difficult task. It often leads to deviations between the final parking posture and the target posture. These deviations can lead to partial occupancy of adjacent parking lots, which poses a safety threat to vehicles parked in these parking lots. However, previous studies have not addressed this issue. In this paper, we aim to evaluate the impact of parking deviation of existing vehicles next to the target parking lot(PDEVNTPL) on the automatic ego vehicle(AEV) parking, in terms of safety, comfort, accuracy, and efficiency of parking. A segmented parking training framework(SPTF) based on soft actor-critic(SAC) is proposed to improve parking performance. In the proposed method, the SAC algorithm incorporates strategy entropy into the objective function, to enable the AEV to learn parking strategies based on a more comprehensive understanding of the environment. Additionally, the SPTF simplifies complex parking tasks to maintain the high performance of deep reinforcement learning(DRL). The experimental results reveal that the PDEVNTPL has a detrimental influence on the AEV parking in terms of safety, accuracy, and comfort, leading to reductions of more than 27%, 54%, and 26%respectively. However, the SAC-based SPTF effectively mitigates this impact, resulting in a considerable increase in the parking success rate from 71% to 93%. Furthermore, the heading angle deviation is significantly reduced from 2.25 degrees to 0.43degrees.

Robustness Assessment of Asynchronous Advantage Actor-Critic Based on Dynamic Skewness and Sparseness Computation: A Parallel Computing View

Reinforcement learning as autonomous learning is greatly driving artificial intelligence(AI)development to practical applications.Having demonstrated the potential to significantly improve synchronously parallel learning,the parallel computing based asynchronous advantage actor-critic(A3C)opens a new door for reinforcement learning.Unfortunately,the acceleration's influence on A3C robustness has been largely overlooked.In this paper,we perform the first robustness assessment of A3C based on parallel computing.By perceiving the policy's action,we construct a global matrix of action probability deviation and define two novel measures of skewness and sparseness to form an integral robustness measure.Based on such static assessment,we then develop a dynamic robustness assessing algorithm through situational whole-space state sampling of changing episodes.Extensive experiments with different combinations of agent number and learning rate are implemented on an A3C-based pathfinding application,demonstrating that our proposed robustness assessment can effectively measure the robustness of A3C,which can achieve an accuracy of 83.3%.

融合Dead-ends和离线监督Actor-Critic的动态治疗策略生成模型

强化学习对数学模型依赖性低,利用经验便于架构和优化模型,非常适合用于动态治疗策略学习。但现有研究仍存在以下问题:1)学习策略最优性的同时未考虑风险,导致学到的策略存在一定的风险;2)忽略了分布偏移问题,导致学到的策略与医生策略完全不同;3)忽略患者的历史观测数据和治疗史,从而不能很好地得到患者状态,进而导致不能学到最优策略。基于此,提出了融合Dead-ends和离线监督Actor-Critic的动态治疗策略生成模型DOSAC-DTR。首先,考虑学到的策略所推荐的治疗行动的风险性,在Actor-Critic框架中融入Dead-ends概念;其次,为缓解分布偏移问题,在Actor-Critic框架中融入医生监督,在最大化预期回报的同时,最小化所学策略与医生策略之间的差距;最后,为了得到包含患者关键历史信息的状态表示,使用基于LSTM的编码器解码器模型对患者的历史观测数据和治疗史进行建模。实验结果表明,DOSAC-DTR相比基线方法有更好的性能,可以得到更低的估计死亡率以及更高的Jaccard系数。

无人机辅助物联网中基于Safe Actor-Critic的信息年龄最小化研究

无人机作为一种新的通信设备,有望在物联网数据采集、监控等业务中发挥关键作用。为保证所采集数据的时效性,利用信息年龄来衡量无人机从物联网设备接收到的数据新鲜度。通过联合优化无人机轨迹和无人机与物联网设备的关联策略以最小化信息年龄加权和,并保证无人机累积飞行能量消耗满足预算要求。由于上述问题同时受短期和长期约束条件的限制,将问题建模为受约束的马尔可夫决策过程(CMDP),并利用Safe Actor-Critic来求解。仿真结果表明,所提算法在最小化信息年龄的同时,能有效保证能量预算。

基于不确定性估计的离线确定型Actor-Critic

Actor-Critic是一种强化学习方法,通过与环境在线试错交互收集样本来学习策略,是求解序贯感知决策问题的有效手段.但是,这种在线交互的主动学习范式在一些复杂真实环境中收集样本时会带来成本和安全问题离线强化学习作为一种基于数据驱动的强化学习范式,强调从静态样本数据集中学习策略,与环境无探索交互,为机器人、自动驾驶、健康护理等真实世界部署应用提供了可行的解决方案,是近年来的研究热点.目前,离线强化学习方法存在学习策略和行为策略之间的分布偏移挑战,针对这个挑战,通常采用策略约束或值函数正则化来限制访问数据集分布之外(Out-Of-Distribution,OOD)的动作,从而导致学习性能过于保守,阻碍了值函数网络的泛化和学习策略的性能提升.为此,本文利用不确定性估计和OOD采样来平衡值函数学习的泛化性和保守性,提出一种基于不确定性估计的离线确定型Actor-Critic方法(Offline Deterministic Actor-Critic based on UncertaintyEstimation,ODACUE).首先,针对确定型策略,给出一种Q值函数的不确定性估计算子定义,理论证明了该算子学到的Q值函数是最优Q值函数的一种悲观估计.然后,将不确定性估计算子应用于确定型Actor-Critic框架中,通过对不确定性估计算子进行凸组合构造Critic学习的目标函数.最后,D4RL基准数据集任务上的实验结果表明:相较于对比算法,ODACUE在11个不同质量等级数据集任务中的总体性能提升最低达9.56%,最高达64.92%.此外,参数分析和消融实验进一步验证了ODACUE的稳定性和泛化能力.

GRU-integrated constrained soft actor-critic learning enabled fully distributed scheduling strategy for residential virtual power plant

In this study,a novel residential virtual power plant(RVPP)scheduling method that leverages a gate recurrent unit(GRU)-integrated deep reinforcement learning(DRL)algorithm is proposed.In the proposed scheme,the GRU-integrated DRL algorithm guides the RVPP to participate effectively in both the day-ahead and real-time markets,lowering the electricity purchase costs and consumption risks for end-users.The Lagrangian relaxation technique is introduced to transform the constrained Markov decision process(CMDP)into an unconstrained optimization problem,which guarantees that the constraints are strictly satisfied without determining the penalty coefficients.Furthermore,to enhance the scalability of the constrained soft actor-critic(CSAC)-based RVPP scheduling approach,a fully distributed scheduling architecture was designed to enable plug-and-play in the residential distributed energy resources(RDER).Case studies performed on the constructed RVPP scenario validated the performance of the proposed methodology in enhancing the responsiveness of the RDER to power tariffs,balancing the supply and demand of the power grid,and ensuring customer comfort.

Actor-critic框架下的二次指派问题求解方法

二次指派问题(QAP)属于NP-hard组合优化问题,在现实生活中有着广泛应用。目前相对成熟的启发式算法通常以问题为导向来设计定制化算法,缺乏迁移泛化能力。为提供一个统一的QAP求解策略,将QAP问题的流量矩阵及距离矩阵抽象成两个无向完全图并构造相应的关联图,从而将设施和地点的指派任务转化为关联图上的节点选择任务,基于actor-critic框架,提出一种全新的求解算法ACQAP。首先,利用多头注意力机制构造策略网络,处理来自图卷积神经网络的节点表征向量;然后,通过actor-critic算法预测每个节点被作为最优节点输出的概率;最后,依据该概率在可行时间内输出满足目标奖励函数的动作决策序列。该算法摆脱人工设计,且适用于不同规模的输入,更加灵活可靠。实验结果表明,在QAPLIB实例上,本算法在精度媲美传统启发式算法的前提下,迁移泛化能力更强;同时相对于NGM等基于学习的算法,求解的指派费用与最优解之间的偏差最小,且在大部分实例中,偏差均小于20%。

Defining Quantum Advantage for Building a Sustainable MVP to Deliver Quantum Computing Services

Quantum Computing (QC) is hailed as the future of computers. After Google’s claim of achieving Quantum Supremacy in 2019, several groups challenged the claim. Some QC experts attribute catastrophic risks that unrestrained QC may cause in the future by collapsing the current cryptographic cybersecurity infrastructure. These predictions are relevant only if QC becomes commercially viable and sustainable in the future. No technology can be a one-way ticket to catastrophe, and neither can the definition of superiority of that technology be. If there are catastrophic risks, large-scale QC can never enter the public domain as a minimum viable product (MVP) unless there are safeguards in place. Those safeguards should obviously become an integral part of the definition of its superiority over the legacy systems. NIST (National Institute of Standards & Technology) is pursuing the standardization of Post Quantum Cryptography (PQC) as that safeguard. However, with all the 82 candidate PQCs failing and companies already offering QC as a service, there’s an urgent need for an alternate strategy to mitigate the impending Q-Day threat and render QC sustainable. Our research proposes a novel encryption-agnostic cybersecurity approach to safeguard QC. It articulates a comprehensive definition of an MVP that can potentially set a sustainable gold standard for defining commercially viable quantum advantage over classical computing.

基于Actor-Critic自适应PID的钢筋套丝头跟踪检测控制系统研究

为适应流水线节奏,不影响生产进程,从而更好地实现钢筋套丝头质量检测和尺寸测量的自动化与智能化,设计了基于同步带直线导轨的钢筋套丝头检测跟踪系统,并提出了一种基于Actor-Critic的自适应PID控制方法,用强化学习的方法根据环境反馈自动调节PID控制器的比例、积分、微分参数。对该方法和其他PID控制方法的响应性能指标进行实验和分析,实验结果表明该方法能够实现高精度、快速响应的跟踪拍摄,保证高精度的套丝头质量检测。

Evaluation of the Clinical Advantages of Laparoscopic Transverse Abdominal Wall Suspension in the Treatment of Pelvic Organ Prolapse

Objective:To evaluate the clinical advantages of laparoscopic transverse abdominal wall suspension in treating pelvic organ prolapse.Methods:Sixty patients diagnosed with moderate to severe pelvic organ prolapse and underwent surgical treatment in our hospital between January 2022 and December 2023 were selected.According to different surgical methods,they were divided into an observation group(given laparoscopic transverse abdominal wall suspension)and a control group(given transvaginal mesh implantation),with 30 subjects/group.The data on perioperative-related indicators,quality of life scores,postoperative recurrence,and complications of the two groups of patients were collected.Results:The postoperative hospitalization days and intraoperative bleeding volume of the observation group were significantly lower than those of the control group,but had longer operation time than that of the control group(P<0.05).The differences between the two groups were statistically significant 6 months after surgery,and the Pelvic Floor Disease Quality of Life Impact Questionnaire(PFIQ-7)score of the observation group was significantly higher than the control group(P<0.05).Both groups of patients completed 12 months of follow-up without any postoperative recurrence.The number of complications in the observation group was slightly lower than that of the control group(P>0.05).Conclusion:Laparoscopic transverse abdominal wall suspension was more effective in treating pelvic organ prolapse and is an ideal surgical procedure.

基于改进Actor-Critic算法的多传感器交叉提示技术

针对在减少战场资源浪费、平衡战场效费比的同时提高目标探测概率,保证目标的可持续跟踪,提出利用改进Actor-Critic算法的多传感器交叉提示技术进行目标探测。首先,综合传感器探测、能耗、时效等因素搭建基于“交叉提示”传感器的动态管理评估模型;其次,重点分析利用Actor-Critic交叉提示算法的传感器管理决策规则,并且提出了Actor-Critic算法,以根据任务自身需求组建中央评价网络,加大传感器与外部环境的交互。仿真结果表明,改进的算法可以加速网络收益,实现对目标的持续性探测,加强传感器之间的交叉提示功能,提升调度的智能化水平,具有较大的应用价值。

广义行为正则化离线Actor-Critic

行为正则化Actor-Critic(BRAC)是一种离线强化学习算法,通过将当前策略与行为策略之间的Kullback-Leibler(KL)散度作为策略目标函数的正则化项来缓解分布偏移问题.但是,由于KL散度是一种无界的分布差异度量,在策略差异过大时,策略目标函数中的累积期望回报项将仅对策略改进发挥有限的作用,从而导致最终学到的策略性能较差.针对该问题,将当前策略与行为策略之间的斜对称Jensen-Shannon(JS)散度作为策略目标函数的正则化项,提出了一种广义行为正则化离线Actor-Critic(GOACBR)算法.理论分析表明:由于斜对称JS散度有界,将其作为正则化项有助于降低策略性能差异.进一步,针对行为策略未知导致难以直接计算当前策略和行为策略间斜对称JS散度的问题,设计了一个辅助网络来对其进行间接估计.最后,给出了GOACBR的收敛性理论证明.在D4RL基准数据集上的评估结果表明:相较于BRAC,GOACBR在所有测试任务上获得的平均累积回报总和提升了289.8%.相关代码公布在https://github.com/houge1996/GOAC.

Comparative Advantage and Latecomer Advantage

To achieve development,developing countries must capitalize on their relative advantages,including comparative advantage and latecomer advantages.A country must develop an industrial structure that is consistent with the comparative advantage of its factor structure in order to avoid the trap of“structural catch-up”that stifles development.The issue for developing countries is that their comparative advantage in core factors of production is unstable and tends to decrease and dissipate.Such a comparative advantage cannot provide adequate growth for convergence to occur.Developed countries,on the other hand,can sustain economic growth due to their comparative advantage in high-quality factors.Latecomer advantage refers to the ability of developing countries to increase their knowledge factor in a cost-effective manner by learning,importing,assimilating,and using knowledge and technology that already exist.Although comparative and latecomer advantages may have a combined effect,each has its distinct characteristics.Latecomer advantage is a special development factor that applies to latecomers.It is crucial at all stages of development,but especially so in the middle-income stage and beyond.Bringing latecomer advantage into full play is essential for developing nations to converge with the advanced countries.

Advantages and Development Prospects of Building Information Modelling(BIM)Technology Application in Highway Engineering

Based on the advantages of the application of Building Information Modelling(BIM)technology in highway engineering,this paper analyzes the difficulties of its in-depth application,grasps the pain points of the application of BIM technology in practice,and approaches from the integration of BIM design-construction.This paper also puts forward the integration of cross-stage information based on the BIM+Geographic Information System(GIS)construction information management system to realize the BIM whole life cycle management,and provides directional support for the continuous and efficient application of BIM technology in engineering practice.

Utilizing Dynamic Data to Quickly Evaluate Advantageous Channels and Applications

In response to the high cost and difficulty of high-speed development and testing data in offshore oil fields, this paper proposes to use the most easily available production performance data as the basis and use the grey correlation method to calculate the correlation coefficient between oil and water wells to characterize the degree of development of advantageous channels. The consistency between the calculated results of this method and the tracer test results is over 80%. Based on the fitting results, the correlation coefficient exceeds 0.74 to determine the existence of an advantageous channel. According to the research results of grey correlation method, Bohai K oilfield has completed the combined profile control and flooding measures, and the daily oil production has increased by 20 m3/d. This method is simple, fast, and can achieve quantitative evaluation, which saves time and investment compared to offshore testing. It has strong application and reference value for the development of other offshore water injection oilfields.

Comparative advantages of chemical compositions of specific edible vegetable oils

As important supplementary to major edible oils, comparative chemical advantages of minor edible oils decidetheir development and usage. In this study, chemical composition of 13 kinds of specific edible vegetable oilswere investigated. The comparative advantages of chemical compositions of these edible oils were obtained asfollows: (1) camellia, tiger nut and almond oil were rich in oleic acid, the contents of which accounted for79.43%, 69.16% and 66.26%, respectively;(2) safflower oil contained the highest content of linoleic acid(76.69%), followed by grape seed (66.85%) and walnut oil (57.30%);(3) perilla seed, siritch, peony seed andherbaceous peony seed oil were rich in α-linolenic acid (59.61%, 43.74%, 40.83% and 30.84%, respectively);(4)the total phytosterol contents of these oils ranged from 91.46 mg/100 g (camellia oil) to 506.46 mg/100 g (siritchoil);and (5) The best source of tocopherols was sacha inchi oil (122.74 mg/100 g), followed by perilla seed oil(55.89 mg/100 g), peony seed oil (53.73 mg/100 g) and herbaceous peony seed oil (47.17 mg/100 g). Thecomparative advantages of these specific edible oils indicated that they possess the high potential nutritionalvalues and health care functions.

Advantages & Disadvantages of Online CAS

Online Computer Algebra Systems (CAS) have become increasingly popular among students and teachers. The reasons are many such as being more flexible, simple to use, accessible from anywhere, etc. However, as with any educational tool, they also have some disadvantages that we should know. The purpose of this study is to analyze advantages and disadvantages of online CASs and propose some techniques to optimize CAS performance in order to reduce weaknesses. The research results reveal that online CAS versions are on the rise but lag in some capabilities in comparison with desktop versions.

Evolution of Growth Model and Cultivation of Competitive Advantages Under the Ambidextrous Innovation Strategy: In the Case of China’s High-Tech Enterprises

This study focuses on the evolution of growth model and cultivation of competitive advantages under the ambidextrous innovation strategy.After a brief introduction of the connotation of ambidextrous innovation strategy,the evolution of growth model of high-tech enterprises under the conventional strategy and the ambidextrous innovation strategy is analyzed.Furthermore,a discussion is made on how to cultivate enterprises’competitive advantages under the ambidextrous innovation strategy,thereby enabling enterprises to stand out from competitors under this new strategic model and truly achieve the goal of sustainable development.

Actor-Critic框架下的数据驱动异步电机离线参数辨识方法

电动汽车用电机的参数辨识可以使电机在任意转速下尽可能输出更高的转矩及效率,是优化电机输出性能的重要手段。传统的基于模型驱动的参数辨识方法的缺点为易受模型误差的影响、抗干扰能力差以及无法实现全转速范围内的转矩最优。鉴于上述缺点,该文研究了一种完全基于实际数据的电动汽车用异步电机离线参数辨识方法,对电机的转子电阻和励磁电感在任意转速下进行了优化,从而使电机能够在特定转速和特定电流下输出最优转矩。为达到电机在特定转速和电流下输出转矩最优的目的,研究了一种基于Actor-Critic框架的电动汽车用异步电机离线参数辨识方法,确定了框架中的观测、奖励和动作的设计。实验证明相对于传统参数辨识方法,该文方法具有更高的精确性和鲁棒性,同时确保了电动汽车用异步电机在任意转速下的输出转矩最优。

Advantage综合录井仪数据采集系统国产化设计

在分析研究Advantage综合录井仪数据采集系统功能与原理的基础上,提出了一种数据采集系统国产化的思路,设计并研制了国产化数据采集系统。新系统从软、硬件两方面着手进行开发,研制了全新的系统软件和设备组成,使Advantage综合录井仪的使用和维护更加稳定、便捷,实现了对原进口设备的替代。