改进Q-Learning的路径规划算法研究

针对Q-Learning算法学习效率低、收敛速度慢且在动态障碍物的环境下路径规划效果不佳的问题,本文提出一种改进Q-Learning的移动机器人路径规划算法.针对该问题,算法根据概率的突变性引入探索因子来平衡探索和利用以加快学习效率;通过在更新函数中设计深度学习因子以保证算法探索概率;融合遗传算法,避免陷入局部路径最优同时按阶段探索最优迭代步长次数,以减少动态地图探索重复率;最后提取输出的最优路径关键节点采用贝塞尔曲线进行平滑处理,进一步保证路径平滑度和可行性.实验通过栅格法构建地图,对比实验结果表明,改进后的算法效率相较于传统算法在迭代次数和路径上均有较大优化,且能够较好的实现动态地图下的路径规划,进一步验证所提方法的有效性和实用性.

改进的Q-learning蜂群算法求解置换流水车间调度问题

针对置换流水车间调度问题,提出了一种基于改进的Q-learning算法的人工蜂群算法。该算法设计了一种改进的奖励函数作为人工蜂群算法的环境,根据奖励函数的优劣来判断下一代种群的寻优策略,并通过Q-learning智能选择人工蜂群算法的蜜源的更新维度数大小,根据选择的维度数大小对编码进行更新,提高了收敛速度和精度,最后使用不同规模的置换流水车间调度问题的实例来验证所提算法的性能,通过对标准实例的计算与其它算法对比,证明该算法的准确性。

基于改进Q-Learning的移动机器人路径规划算法

随着移动机器人在生产生活中的深入应用,其路径规划能力也需要向快速性和环境适应性兼备发展。为解决现有移动机器人使用强化学习方法进行路径规划时存在的探索前期容易陷入局部最优、反复搜索同一区域,探索后期收敛率低、收敛速度慢的问题,本研究提出一种改进的Q-Learning算法。该算法改进Q矩阵赋值方法,使迭代前期探索过程具有指向性,并降低碰撞的情况;改进Q矩阵迭代方法,使Q矩阵更新具有前瞻性,避免在一个小区域中反复探索;改进随机探索策略,在迭代前期全面利用环境信息,后期向目标点靠近。在不同栅格地图仿真验证结果表明,本文算法在Q-Learning算法的基础上,通过上述改进降低探索过程中的路径长度、减少抖动并提高收敛的速度,具有更高的计算效率。

基于Q-Learning的航空器滑行路径规划研究

针对传统算法规划航空器滑行路径准确度低、不能根据整体场面运行情况进行路径规划的问题,提出一种基于Q-Learning的路径规划方法。通过对机场飞行区网络结构模型和强化学习的仿真环境分析,设置了状态空间和动作空间,并根据路径的合规性和合理性设定了奖励函数,将路径合理性评价值设置为滑行路径长度与飞行区平均滑行时间乘积的倒数。最后,分析了动作选择策略参数对路径规划模型的影响。结果表明,与A*算法和Floyd算法相比,基于Q-Learning的路径规划在滑行距离最短的同时,避开了相对繁忙的区域,路径合理性评价值高。

基于Q-Learning的分簇无线传感网信任管理机制

针对无线传感器网络中存在的安全问题,提出了基于Q-Learning的分簇无线传感网信任管理机制(Q-learning based trust management mechanism for clustered wireless sensor networks,QLTMM-CWSN).该机制主要考虑通信信任、数据信任和能量信任3个方面.在网络运行过程中,基于节点的通信行为、数据分布和能量消耗,使用Q-Learning算法更新节点信任值,并选择簇内信任值最高的节点作为可信簇头节点.当簇中主簇头节点的信任值低于阈值时,可信簇头节点代替主簇头节点管理簇内成员节点,维护正常的数据传输.研究结果表明,QLTMM-CWSN机制能有效抵御通信攻击、伪造本地数据攻击、能量攻击和混合攻击.

基于多步信息辅助的Q-learning路径规划算法

为提升静态环境下移动机器人路径规划能力,解决传统Q-learning算法在路径规划中收敛速度慢的问题,提出一种基于多步信息辅助机制的Q-learning改进算法。利用ε-greedy策略中贪婪动作的多步信息与历史最优路径长度更新资格迹,使有效的资格迹在算法迭代中持续发挥作用,用保存的多步信息解决可能落入的循环陷阱;使用局部多花朵的花授粉算法初始化Q值表,提升机器人前期搜索效率;基于机器人不同探索阶段的目的,结合迭代路径长度的标准差与机器人成功到达目标点的次数设计动作选择策略,以增强算法对环境信息探索与利用的平衡能力。实验结果表明:该算法具有较快的收敛速度,验证了算法的可行性与有效性。

基于softmax的加权Double Q-Learning算法

强化学习作为机器学习的一个分支,用于描述和解决智能体在与环境的交互过程中,通过学习策略以达成回报最大化的问题。Q-Learning作为无模型强化学习的经典方法,存在过估计引起的最大化偏差问题,并且在环境中奖励存在噪声时表现不佳。Double Q-Learning(DQL)的出现解决了过估计问题,但同时造成了低估问题。为解决以上算法的高低估问题,提出了基于softmax的加权Q-Learning算法,并将其与DQL相结合,提出了一种新的基于softmax的加权Double Q-Learning算法(WDQL-Softmax)。该算法基于加权双估计器的构造,对样本期望值进行softmax操作得到权重,使用权重估计动作价值,有效平衡对动作价值的高估和低估问题,使估计值更加接近理论值。实验结果表明,在离散动作空间中,相比于Q-Learning算法、DQL算法和WDQL算法,WDQL-Softmax算法的收敛速度更快且估计值与理论值的误差更小。

An Underwater Target Detection Algorithm Based on Attention Mechanism and Improved YOLOv7

For underwater robots in the process of performing target detection tasks,the color distortion and the uneven quality of underwater images lead to great difficulties in the feature extraction process of the model,which is prone to issues like error detection,omission detection,and poor accuracy.Therefore,this paper proposed the CER-YOLOv7(CBAM-EIOU-RepVGG-YOLOv7)underwater target detection algorithm.To improve the algorithm’s capability to retain valid features from both spatial and channel perspectives during the feature extraction phase,we have added a Convolutional Block Attention Module(CBAM)to the backbone network.The Reparameterization Visual Geometry Group(RepVGG)module is inserted into the backbone to improve the training and inference capabilities.The Efficient Intersection over Union(EIoU)loss is also used as the localization loss function,which reduces the error detection rate and missed detection rate of the algorithm.The experimental results of the CER-YOLOv7 algorithm on the UPRC(Underwater Robot Prototype Competition)dataset show that the mAP(mean Average Precision)score of the algorithm is 86.1%,which is a 2.2%improvement compared to the YOLOv7.The feasibility and validity of the CER-YOLOv7 are proved through ablation and comparison experiments,and it is more suitable for underwater target detection.

多无人机辅助边缘计算场景下基于Q-learning的任务卸载优化

引入多无人机辅助边缘计算系统,由多个无人机和原有边缘服务器共同为移动用户提供通信和计算资源;将优化问题建模为资源竞争和卸载决策约束下的系统总效用最大化问题,系统总效用由用户满意度、任务延迟和系统能耗3个因素组成.由于优化模型是一个具有NP难属性的非凸问题,故采用强化学习方法求解得到使系统总效用最大的最优任务卸载决策集.仿真实验结果表明,与贪心顺序调优卸载方案和随机选择卸载方案相比,该文提出的Q-learning方案的系统总效用分别提高了15%和43%以上.

基于改进Q-learning算法移动机器人局部路径研究

针对局部路径规划时因无法提前获取环境信息导致移动机器人搜索不到合适的路径,以及在采用马尔可夫决策过程中传统强化学习算法应用于局部路径规划时存在着学习效率低下及收敛速度较慢等问题,提出一种改进的Q-learn-ing(QL)算法。首先设计一种动态自适应贪婪策略,用于平衡移动机器人对环境探索和利用之间的问题;其次根据A*算法思想设计启发式学习评估模型,从而动态调整学习因子并为搜索路径提供导向作用;最后引入三阶贝塞尔曲线规划对路径进行平滑处理。通过Pycharm平台仿真结果表明,使得改进后的QL算法所规划的路径长度、搜索效率及路径平滑性等特性上都优于传统Sarsa算法及QL算法,比传统Sarsa算法迭代次数提高32.3%,搜索时间缩短27.08%,比传统QL算法迭代次数提高27.32%,搜索时间缩短17.28%,路径规划的拐点大幅度减少,局部路径优化效果较为明显。

一种基于Q-learning强化学习的导向性处理器安全性模糊测试方案

针对处理器安全性模糊测试在进行细粒度变异时遗传算法存在一定的盲目性,易使生成的测试用例触发相同类型漏洞的问题,提出了一种基于Q-learning强化学习的导向性处理器安全性模糊测试方案。通过测试用例的状态值和所触发的漏洞类型对应的权值构造奖励函数,使用强化学习指导生成具有针对性和导向性的测试用例,快速地触发不同类型的漏洞。在Hikey970平台上的实验验证了基于ARMv8的测试用例生成框架的有效性,并且相较于传统使用遗传算法作为反馈的策略,本文方案在相同时间内生成有效测试用例的的数量多19.15%,发现漏洞类型的数量多80.00%。

基于改进Q-learning算法的移动机器人路径规划

针对传统Q-learning算法应用在路径规划中存在收敛速度慢、运行时间长、学习效率差等问题,提出一种将人工势场法和传统Q-learning算法结合的改进Q-learning算法。该算法引入人工势场法的引力函数与斥力函数,通过对比引力函数动态选择奖励值,以及对比斥力函数计算姿值,动态更新Q值,使移动机器人具有目的性的探索,并且优先选择离障碍物较远的位置移动。通过仿真实验证明,与传统Q-learning算法、引入引力场算法对比,改进Q-learning算法加快了收敛速度,缩短了运行时间,提高了学习效率,降低了与障碍物相撞的概率,使移动机器人能够快速地找到一条无碰撞通路。

Q-Learning-Assisted Meta-Heuristics for Scheduling Distributed Hybrid Flow Shop Problems

The flow shop scheduling problem is important for the manufacturing industry.Effective flow shop scheduling can bring great benefits to the industry.However,there are few types of research on Distributed Hybrid Flow Shop Problems(DHFSP)by learning assisted meta-heuristics.This work addresses a DHFSP with minimizing the maximum completion time(Makespan).First,a mathematical model is developed for the concerned DHFSP.Second,four Q-learning-assisted meta-heuristics,e.g.,genetic algorithm(GA),artificial bee colony algorithm(ABC),particle swarm optimization(PSO),and differential evolution(DE),are proposed.According to the nature of DHFSP,six local search operations are designed for finding high-quality solutions in local space.Instead of randomselection,Q-learning assists meta-heuristics in choosing the appropriate local search operations during iterations.Finally,based on 60 cases,comprehensive numerical experiments are conducted to assess the effectiveness of the proposed algorithms.The experimental results and discussions prove that using Q-learning to select appropriate local search operations is more effective than the random strategy.To verify the competitiveness of the Q-learning assistedmeta-heuristics,they are compared with the improved iterated greedy algorithm(IIG),which is also for solving DHFSP.The Friedman test is executed on the results by five algorithms.It is concluded that the performance of four Q-learning-assisted meta-heuristics are better than IIG,and the Q-learning-assisted PSO shows the best competitiveness.

IGED:Towards Intelligent DDoS Detection Model Using Improved Generalized Entropy and DNN

As the scale of the networks continually expands,the detection of distributed denial of service(DDoS)attacks has become increasingly vital.We propose an intelligent detection model named IGED by using improved generalized entropy and deep neural network(DNN).The initial detection is based on improved generalized entropy to filter out as much normal traffic as possible,thereby reducing data volume.Then the fine detection is based on DNN to perform precise DDoS detection on the filtered suspicious traffic,enhancing the neural network’s generalization capabilities.Experimental results show that the proposed method can efficiently distinguish normal traffic from DDoS traffic.Compared with the benchmark methods,our method reaches 99.9%on low-rate DDoS(LDDoS),flooded DDoS and CICDDoS2019 datasets in terms of both accuracy and efficiency in identifying attack flows while reducing the time by 17%,31%and 8%.

Value of improved nursing measures and enhanced nursing management to reduce the occurrence of adverse events in pediatric infusion

BACKGROUND Intravenous infusion is a common method of drug administration in clinical practice.Errors in any aspect of the infusion process,from the verification of medical orders,preparation of the drug solution,to infusion by nursing staff,may cause adverse infusion events.AIM To analyzed the value of improving nursing measures and enhancing nursing management to reduce the occurrence of adverse events in pediatric infusion.METHODS The clinical data of 130 children who received an infusion in the pediatric department of our hospital from May 2020 to May 2021 were analyzed and divided into two groups according to the differences in nursing measures and nursing management:65 patients in the control group received conventional nursing and nursing management interventions,while 65 patients in the observation group received improved nursing measure interventions and enhanced nursing management.The occurrence of adverse events,compliance of children,satisfaction of children’s families,and complaints regarding the transfusion treatment were recorded in both groups.RESULTS The incidence of fluid extravasation and infusion set dislodgement in the observation group were 3.08%and 1.54%,respectively,which were significantly lower than 12.31%and 13.85%in the control group(P<0.05),while repeated punctures and medication addition errors in the observation group were 3.08%and 0.00%,respectively,which were lower than 9.23%and 3.08%in the control group,but there was no significant difference(P>0.05).The compliance rate of children in the observation group was 98.46%(64/65),which was significantly higher than 87.69%(57/65)in the control group,and the satisfaction rate of children’s families was 96.92%(63/65),which was significantly higher than 86.15%(56/65)in the control group(P<0.05).The observation group did not receive any complaints from the child’s family,whereas the control group received four complaints,two of which were due to the crying of the child caused by repeated punctures,one due to the poor attitude of the nurse,and one due to medication addition errors,with a cumulative complaint rate of 6.15%.The cumulative complaint rate of the observation group was significantly lower than that of the control group(P<0.05).CONCLUSION Improving nursing measures and enhancing nursing management can reduce the incidence of fluid extravasation and infusion set dislodgement in pediatric patients,improve children’s compliance and satisfaction of their families,and reduce family complaints.

Traffic Control Based on Integrated Kalman Filtering and Adaptive Quantized Q-Learning Framework for Internet of Vehicles

Intelligent traffic control requires accurate estimation of the road states and incorporation of adaptive or dynamically adjusted intelligent algorithms for making the decision.In this article,these issues are handled by proposing a novel framework for traffic control using vehicular communications and Internet of Things data.The framework integrates Kalman filtering and Q-learning.Unlike smoothing Kalman filtering,our data fusion Kalman filter incorporates a process-aware model which makes it superior in terms of the prediction error.Unlike traditional Q-learning,our Q-learning algorithm enables adaptive state quantization by changing the threshold of separating low traffic from high traffic on the road according to the maximum number of vehicles in the junction roads.For evaluation,the model has been simulated on a single intersection consisting of four roads:east,west,north,and south.A comparison of the developed adaptive quantized Q-learning(AQQL)framework with state-of-the-art and greedy approaches shows the superiority of AQQL with an improvement percentage in terms of the released number of vehicles of AQQL is 5%over the greedy approach and 340%over the state-of-the-art approach.Hence,AQQL provides an effective traffic control that can be applied in today’s intelligent traffic system.

Novel Static Security and Stability Control of Power Systems Based on Artificial Emotional Lazy Q-Learning

The stability problem of power grids has become increasingly serious in recent years as the size of novel power systems increases.In order to improve and ensure the stable operation of the novel power system,this study proposes an artificial emotional lazy Q-learning method,which combines artificial emotion,lazy learning,and reinforcement learning for static security and stability analysis of power systems.Moreover,this study compares the analysis results of the proposed method with those of the small disturbance method for a stand-alone power system and verifies that the proposed lazy Q-learning method is able to effectively screen useful data for learning,and improve the static security stability of the new type of power system more effectively than the traditional proportional-integral-differential control and Q-learning methods.

An Improved JSO and Its Application in Spreader Optimization of Large Span Corridor Bridge

In this paper,given the shortcomings of jellyfish search algorithmwith low search ability in the early stage and easy to fall into local optimal solution,this paper introduces adaptive weight function and elite strategy,improving the global search scope in the early stage and the ability to refine the local development in the later stage.In the numerical study,the benchmark problem of dimensional optimization with a 10-bar truss structure and simultaneous dimensional shape optimization with a 15-bar truss structure is adopted,and the corresponding penalty method is used for constraint treatment.The test results show that the improved jellyfish search algorithm can provide better truss sections as well as weights.Because when the steel main truss of the large-span covered bridge is lifted,the site is limited and the large lifting equipment cannot enter the site,and the original structure does not meet the problem of stress concentration and large deformation of the bolt group,so the spreader is used to lift,and the improved jellyfish search algorithm is introduced into the design optimization of the spreader.The results show that the improved jellyfish algorithm can efficiently and accurately find out the optimal shape and weight of the spreader,and throughMidas Civil simulation,the spreader used canmeet the requirements of weight and safety.

An improved typhoon monitoring model based on precipitable water vapor and pressure

The potential of monitoring the movement of typhoons using the precipitable water vapor(PWV) has been confirmed. However, monitoring the movement of typhoon is focused on PWV, making it difficult to describe the movement of a typhoon in detail minutely and resulting in insufficient accuracy. Hence,based on PWV and meteorological data, we propose an improved typhoon monitoring mode. First, the European Centre for Medium-Range Weather Forecasts Reanalysis 5-derived PWV(ERA5-PWV) and the Global Navigation Satellite System-derived PWV(GNSS-PWV) were compared with the reference radiosonde PWV(RS-PWV). Then, using the PWV and atmospheric parameters derived from ERA5, we discussed the anomalous variations of PWV, pressure(P), precipitation, and wind speed during different typhoons. Finally, we compiled a list of critical factors related to typhoon movement, PWV and P. We developed an improved multi-factor typhoon monitoring mode(IMTM) with different models(i.e.,IMTM-I and IMTM-II) in different cases with a higher density of GNSS observation or only Numerical Weather Prediction(NWP) data. The IMTM was evaluated through the reference movement speeds of HATO and Mangkhut from the China Meteorological Observatory Typhoon Network(CMOTN). The results show that the root mean square(RMS) of the IMTM-I is 1.26 km/h based on ERA5-P and ERA5-PWV,and the absolute bias values are mostly within 2 km/h. Compared with the models considering the single factor ERA5-P/ERA5-PWV, the RMS of the IMTM-I is improved by 26.3% and 38.5%, respectively. The IMTM-II model manifests a residual of only 0.35 km/h. Compared with the single-factor model based on GNSS-PWV/P, the residual of the IMTM-II model is reduced by 90.8% and 84.1%, respectively. These results propose that the typhoon movement monitoring approach combining PWV and P has evident advantages over the single-factor model and is expected to supplement traditional typhoon monitoring.

Energy-Saving Distributed Flexible Job Shop Scheduling Optimization with Dual Resource Constraints Based on Integrated Q-Learning Multi-Objective Grey Wolf Optimizer

The distributed flexible job shop scheduling problem(DFJSP)has attracted great attention with the growth of the global manufacturing industry.General DFJSP research only considers machine constraints and ignores worker constraints.As one critical factor of production,effective utilization of worker resources can increase productivity.Meanwhile,energy consumption is a growing concern due to the increasingly serious environmental issues.Therefore,the distributed flexible job shop scheduling problem with dual resource constraints(DFJSP-DRC)for minimizing makespan and total energy consumption is studied in this paper.To solve the problem,we present a multi-objective mathematical model for DFJSP-DRC and propose a Q-learning-based multi-objective grey wolf optimizer(Q-MOGWO).In Q-MOGWO,high-quality initial solutions are generated by a hybrid initialization strategy,and an improved active decoding strategy is designed to obtain the scheduling schemes.To further enhance the local search capability and expand the solution space,two wolf predation strategies and three critical factory neighborhood structures based on Q-learning are proposed.These strategies and structures enable Q-MOGWO to explore the solution space more efficiently and thus find better Pareto solutions.The effectiveness of Q-MOGWO in addressing DFJSP-DRC is verified through comparison with four algorithms using 45 instances.The results reveal that Q-MOGWO outperforms comparison algorithms in terms of solution quality.