改进Q-Learning的路径规划算法研究

针对Q-Learning算法学习效率低、收敛速度慢且在动态障碍物的环境下路径规划效果不佳的问题,本文提出一种改进Q-Learning的移动机器人路径规划算法.针对该问题,算法根据概率的突变性引入探索因子来平衡探索和利用以加快学习效率;通过在更新函数中设计深度学习因子以保证算法探索概率;融合遗传算法,避免陷入局部路径最优同时按阶段探索最优迭代步长次数,以减少动态地图探索重复率;最后提取输出的最优路径关键节点采用贝塞尔曲线进行平滑处理,进一步保证路径平滑度和可行性.实验通过栅格法构建地图,对比实验结果表明,改进后的算法效率相较于传统算法在迭代次数和路径上均有较大优化,且能够较好的实现动态地图下的路径规划,进一步验证所提方法的有效性和实用性.

基于改进Q-Learning的移动机器人路径规划算法

随着移动机器人在生产生活中的深入应用,其路径规划能力也需要向快速性和环境适应性兼备发展。为解决现有移动机器人使用强化学习方法进行路径规划时存在的探索前期容易陷入局部最优、反复搜索同一区域,探索后期收敛率低、收敛速度慢的问题,本研究提出一种改进的Q-Learning算法。该算法改进Q矩阵赋值方法,使迭代前期探索过程具有指向性,并降低碰撞的情况;改进Q矩阵迭代方法,使Q矩阵更新具有前瞻性,避免在一个小区域中反复探索;改进随机探索策略,在迭代前期全面利用环境信息,后期向目标点靠近。在不同栅格地图仿真验证结果表明,本文算法在Q-Learning算法的基础上,通过上述改进降低探索过程中的路径长度、减少抖动并提高收敛的速度,具有更高的计算效率。

基于Q-Learning的航空器滑行路径规划研究

针对传统算法规划航空器滑行路径准确度低、不能根据整体场面运行情况进行路径规划的问题,提出一种基于Q-Learning的路径规划方法。通过对机场飞行区网络结构模型和强化学习的仿真环境分析,设置了状态空间和动作空间,并根据路径的合规性和合理性设定了奖励函数,将路径合理性评价值设置为滑行路径长度与飞行区平均滑行时间乘积的倒数。最后,分析了动作选择策略参数对路径规划模型的影响。结果表明,与A*算法和Floyd算法相比,基于Q-Learning的路径规划在滑行距离最短的同时,避开了相对繁忙的区域,路径合理性评价值高。

基于Q-Learning的分簇无线传感网信任管理机制

针对无线传感器网络中存在的安全问题,提出了基于Q-Learning的分簇无线传感网信任管理机制(Q-learning based trust management mechanism for clustered wireless sensor networks,QLTMM-CWSN).该机制主要考虑通信信任、数据信任和能量信任3个方面.在网络运行过程中,基于节点的通信行为、数据分布和能量消耗,使用Q-Learning算法更新节点信任值,并选择簇内信任值最高的节点作为可信簇头节点.当簇中主簇头节点的信任值低于阈值时,可信簇头节点代替主簇头节点管理簇内成员节点,维护正常的数据传输.研究结果表明,QLTMM-CWSN机制能有效抵御通信攻击、伪造本地数据攻击、能量攻击和混合攻击.

基于多步信息辅助的Q-learning路径规划算法

为提升静态环境下移动机器人路径规划能力,解决传统Q-learning算法在路径规划中收敛速度慢的问题,提出一种基于多步信息辅助机制的Q-learning改进算法。利用ε-greedy策略中贪婪动作的多步信息与历史最优路径长度更新资格迹,使有效的资格迹在算法迭代中持续发挥作用,用保存的多步信息解决可能落入的循环陷阱;使用局部多花朵的花授粉算法初始化Q值表,提升机器人前期搜索效率;基于机器人不同探索阶段的目的,结合迭代路径长度的标准差与机器人成功到达目标点的次数设计动作选择策略,以增强算法对环境信息探索与利用的平衡能力。实验结果表明:该算法具有较快的收敛速度,验证了算法的可行性与有效性。

基于softmax的加权Double Q-Learning算法

强化学习作为机器学习的一个分支,用于描述和解决智能体在与环境的交互过程中,通过学习策略以达成回报最大化的问题。Q-Learning作为无模型强化学习的经典方法,存在过估计引起的最大化偏差问题,并且在环境中奖励存在噪声时表现不佳。Double Q-Learning(DQL)的出现解决了过估计问题,但同时造成了低估问题。为解决以上算法的高低估问题,提出了基于softmax的加权Q-Learning算法,并将其与DQL相结合,提出了一种新的基于softmax的加权Double Q-Learning算法(WDQL-Softmax)。该算法基于加权双估计器的构造,对样本期望值进行softmax操作得到权重,使用权重估计动作价值,有效平衡对动作价值的高估和低估问题,使估计值更加接近理论值。实验结果表明,在离散动作空间中,相比于Q-Learning算法、DQL算法和WDQL算法,WDQL-Softmax算法的收敛速度更快且估计值与理论值的误差更小。

Screening biomarkers for spinal cord injury using weighted gene co-expression network analysis and machine learning

Immune changes and inflammatory responses have been identified as central events in the pathological process of spinal co rd injury.They can greatly affect nerve regeneration and functional recovery.However,there is still limited understanding of the peripheral immune inflammato ry response in spinal cord inju ry.In this study.we obtained microRNA expression profiles from the peripheral blood of patients with spinal co rd injury using high-throughput sequencing.We also obtained the mRNA expression profile of spinal cord injury patients from the Gene Expression Omnibus(GEO)database(GSE151371).We identified 54 differentially expressed microRNAs and 1656 diffe rentially expressed genes using bioinformatics approaches.Functional enrichment analysis revealed that various common immune and inflammation-related signaling pathways,such as neutrophil extracellular trap formation pathway,T cell receptor signaling pathway,and nuclear factor-κB signal pathway,we re abnormally activated or inhibited in spinal cord inju ry patient samples.We applied an integrated strategy that combines weighted gene co-expression network analysis,LASSO logistic regression,and SVM-RFE algorithm and identified three biomarke rs associated with spinal cord injury:ANO10,BST1,and ZFP36L2.We verified the expression levels and diagnostic perfo rmance of these three genes in the original training dataset and clinical samples through the receiver operating characteristic curve.Quantitative polymerase chain reaction results showed that ANO20 and BST1 mRNA levels were increased and ZFP36L2 mRNA was decreased in the peripheral blood of spinal cord injury patients.We also constructed a small RNA-mRNA interaction network using Cytoscape.Additionally,we evaluated the proportion of 22 types of immune cells in the peripheral blood of spinal co rd injury patients using the CIBERSORT tool.The proportions of naive B cells,plasma cells,monocytes,and neutrophils were increased while the proportions of memory B cells,CD8^(+)T cells,resting natural killer cells,resting dendritic cells,and eosinophils were markedly decreased in spinal cord injury patients increased compared with healthy subjects,and ANO10,BST1 and ZFP26L2we re closely related to the proportion of certain immune cell types.The findings from this study provide new directions for the development of treatment strategies related to immune inflammation in spinal co rd inju ry and suggest that ANO10,BST2,and ZFP36L2 are potential biomarkers for spinal cord injury.The study was registe red in the Chinese Clinical Trial Registry(registration No.ChiCTR2200066985,December 12,2022).

基于改进Q-learning算法移动机器人局部路径研究

针对局部路径规划时因无法提前获取环境信息导致移动机器人搜索不到合适的路径,以及在采用马尔可夫决策过程中传统强化学习算法应用于局部路径规划时存在着学习效率低下及收敛速度较慢等问题,提出一种改进的Q-learn-ing(QL)算法。首先设计一种动态自适应贪婪策略,用于平衡移动机器人对环境探索和利用之间的问题;其次根据A*算法思想设计启发式学习评估模型,从而动态调整学习因子并为搜索路径提供导向作用;最后引入三阶贝塞尔曲线规划对路径进行平滑处理。通过Pycharm平台仿真结果表明,使得改进后的QL算法所规划的路径长度、搜索效率及路径平滑性等特性上都优于传统Sarsa算法及QL算法,比传统Sarsa算法迭代次数提高32.3%,搜索时间缩短27.08%,比传统QL算法迭代次数提高27.32%,搜索时间缩短17.28%,路径规划的拐点大幅度减少,局部路径优化效果较为明显。

一种基于Q-learning强化学习的导向性处理器安全性模糊测试方案

针对处理器安全性模糊测试在进行细粒度变异时遗传算法存在一定的盲目性,易使生成的测试用例触发相同类型漏洞的问题,提出了一种基于Q-learning强化学习的导向性处理器安全性模糊测试方案。通过测试用例的状态值和所触发的漏洞类型对应的权值构造奖励函数,使用强化学习指导生成具有针对性和导向性的测试用例,快速地触发不同类型的漏洞。在Hikey970平台上的实验验证了基于ARMv8的测试用例生成框架的有效性,并且相较于传统使用遗传算法作为反馈的策略,本文方案在相同时间内生成有效测试用例的的数量多19.15%,发现漏洞类型的数量多80.00%。

Q-Learning-Assisted Meta-Heuristics for Scheduling Distributed Hybrid Flow Shop Problems

The flow shop scheduling problem is important for the manufacturing industry.Effective flow shop scheduling can bring great benefits to the industry.However,there are few types of research on Distributed Hybrid Flow Shop Problems(DHFSP)by learning assisted meta-heuristics.This work addresses a DHFSP with minimizing the maximum completion time(Makespan).First,a mathematical model is developed for the concerned DHFSP.Second,four Q-learning-assisted meta-heuristics,e.g.,genetic algorithm(GA),artificial bee colony algorithm(ABC),particle swarm optimization(PSO),and differential evolution(DE),are proposed.According to the nature of DHFSP,six local search operations are designed for finding high-quality solutions in local space.Instead of randomselection,Q-learning assists meta-heuristics in choosing the appropriate local search operations during iterations.Finally,based on 60 cases,comprehensive numerical experiments are conducted to assess the effectiveness of the proposed algorithms.The experimental results and discussions prove that using Q-learning to select appropriate local search operations is more effective than the random strategy.To verify the competitiveness of the Q-learning assistedmeta-heuristics,they are compared with the improved iterated greedy algorithm(IIG),which is also for solving DHFSP.The Friedman test is executed on the results by five algorithms.It is concluded that the performance of four Q-learning-assisted meta-heuristics are better than IIG,and the Q-learning-assisted PSO shows the best competitiveness.

Traffic Control Based on Integrated Kalman Filtering and Adaptive Quantized Q-Learning Framework for Internet of Vehicles

Intelligent traffic control requires accurate estimation of the road states and incorporation of adaptive or dynamically adjusted intelligent algorithms for making the decision.In this article,these issues are handled by proposing a novel framework for traffic control using vehicular communications and Internet of Things data.The framework integrates Kalman filtering and Q-learning.Unlike smoothing Kalman filtering,our data fusion Kalman filter incorporates a process-aware model which makes it superior in terms of the prediction error.Unlike traditional Q-learning,our Q-learning algorithm enables adaptive state quantization by changing the threshold of separating low traffic from high traffic on the road according to the maximum number of vehicles in the junction roads.For evaluation,the model has been simulated on a single intersection consisting of four roads:east,west,north,and south.A comparison of the developed adaptive quantized Q-learning(AQQL)framework with state-of-the-art and greedy approaches shows the superiority of AQQL with an improvement percentage in terms of the released number of vehicles of AQQL is 5%over the greedy approach and 340%over the state-of-the-art approach.Hence,AQQL provides an effective traffic control that can be applied in today’s intelligent traffic system.

Optimal investment based on relative performance and weighted utility

This paper studies the optimal portfolio allocation of a fund manager when he bases decisions on both the absolute level of terminal relative performance and the change value of terminal relative performance comparison to a predefined reference point. We find the optimal investment strategy by maximizing a weighted average utility of a concave utility and an Sshaped utility via a concavification technique and the martingale method. Numerical results are carried out to show the impact of the extent to which the manager pays attention to the change of relative performance related to the reference point on the optimal terminal relative performance.

Novel Static Security and Stability Control of Power Systems Based on Artificial Emotional Lazy Q-Learning

The stability problem of power grids has become increasingly serious in recent years as the size of novel power systems increases.In order to improve and ensure the stable operation of the novel power system,this study proposes an artificial emotional lazy Q-learning method,which combines artificial emotion,lazy learning,and reinforcement learning for static security and stability analysis of power systems.Moreover,this study compares the analysis results of the proposed method with those of the small disturbance method for a stand-alone power system and verifies that the proposed lazy Q-learning method is able to effectively screen useful data for learning,and improve the static security stability of the new type of power system more effectively than the traditional proportional-integral-differential control and Q-learning methods.

Energy-Saving Distributed Flexible Job Shop Scheduling Optimization with Dual Resource Constraints Based on Integrated Q-Learning Multi-Objective Grey Wolf Optimizer

The distributed flexible job shop scheduling problem(DFJSP)has attracted great attention with the growth of the global manufacturing industry.General DFJSP research only considers machine constraints and ignores worker constraints.As one critical factor of production,effective utilization of worker resources can increase productivity.Meanwhile,energy consumption is a growing concern due to the increasingly serious environmental issues.Therefore,the distributed flexible job shop scheduling problem with dual resource constraints(DFJSP-DRC)for minimizing makespan and total energy consumption is studied in this paper.To solve the problem,we present a multi-objective mathematical model for DFJSP-DRC and propose a Q-learning-based multi-objective grey wolf optimizer(Q-MOGWO).In Q-MOGWO,high-quality initial solutions are generated by a hybrid initialization strategy,and an improved active decoding strategy is designed to obtain the scheduling schemes.To further enhance the local search capability and expand the solution space,two wolf predation strategies and three critical factory neighborhood structures based on Q-learning are proposed.These strategies and structures enable Q-MOGWO to explore the solution space more efficiently and thus find better Pareto solutions.The effectiveness of Q-MOGWO in addressing DFJSP-DRC is verified through comparison with four algorithms using 45 instances.The results reveal that Q-MOGWO outperforms comparison algorithms in terms of solution quality.

基于Q-Learning的动态BLE Mesh网络高能效路由算法

针对动态低功耗蓝牙(Bluetooth Low Energy, BLE)Mesh网络规范采用的管理式泛洪路由机制所导致的数据包冗余和高能耗的问题,提出了动态BLE Mesh网络高能效路由算法。通过建立基于Q-Learning的BLE Mesh网络路由模型,将BLE Mesh网络中节点的剩余能量、转发成本、移动因子以及接收信号强度指示(Received Signal Strength Indicator, RSSI)纳入到奖励函数的设计中。同时对BLE Mesh网络中的心跳机制进行改进,使网络中的节点更加高效地获取环境信息。采用基于探索的路由发现机制来更新网络中节点的Q值,使其能更加准确反映节点移动所导致的网络拓扑变化。仿真结果表明,该算法在节点能耗和网络开销上均优于传统的管理式泛洪路由机制。

The Weighted Basis for PHT-Splines

PHT-splines are defined as polynomial splines over hierarchical T-meshes with very efficient local refinement properties.The original PHT-spline basis functions constructed by the truncation mechanism have a decay phenomenon,resulting in numerical instability.The non-decay basis functions are constructed as the B-splines that are defined on the 2×2 tensor product meshes associated with basis vertices in Kang et al.,but at the cost of losing the partition of unity.In the field of finite element analysis and topology optimization,forming the partition of unity is the default ingredient for constructing basis functions of approximate spaces.In this paper,we will show that the non-decay PHT-spline basis functions proposed by Kang et al.can be appropriately modified to form a partition of unity.Each non-decay basis function is multiplied by a positive weight to form the weighted basis.The weights are solved such that the sum of weighted bases is equal to 1 on the domain.We provide two methods for calculatingweights,based on geometric information of basis functions and the subdivision of PHT-splines.Weights are given in the form of explicit formulas and can be efficiently calculated.We also prove that the weights on the admissible hierarchical T-meshes are positive.

Geographically and Temporally Weighted Regression in Assessing Dengue Fever Spread Factors in Yunnan Border Regions

Objective This study employs the Geographically and Temporally Weighted Regression(GTWR)model to assess the impact of meteorological elements and imported cases on dengue fever outbreaks,emphasizing the spatial-temporal variability of these factors in border regions.Methods We conducted a descriptive analysis of dengue fever’s temporal-spatial distribution in Yunnan border areas.Utilizing annual data from 2013 to 2019,with each county in the Yunnan border serving as a spatial unit,we constructed a GTWR model to investigate the determinants of dengue fever and their spatio-temporal heterogeneity in this region.Results The GTWR model,proving more effective than Ordinary Least Squares(OLS)analysis,identified significant spatial and temporal heterogeneity in factors influencing dengue fever’s spread along the Yunnan border.Notably,the GTWR model revealed a substantial variation in the relationship between indigenous dengue fever incidence,meteorological variables,and imported cases across different counties.Conclusion In the Yunnan border areas,local dengue incidence is affected by temperature,humidity,precipitation,wind speed,and imported cases,with these factors’influence exhibiting notable spatial and temporal variation.

An Improved Solov2 Based on Attention Mechanism and Weighted Loss Function for Electrical Equipment Instance Segmentation

The current existing problem of deep learning framework for the detection and segmentation of electrical equipment is dominantly related to low precision.Because of the reliable,safe and easy-to-operate technology provided by deep learning-based video surveillance for unmanned inspection of electrical equipment,this paper uses the bottleneck attention module(BAM)attention mechanism to improve the Solov2 model and proposes a new electrical equipment segmentation mode.Firstly,the BAM attention mechanism is integrated into the feature extraction network to adaptively learn the correlation between feature channels,thereby improving the expression ability of the feature map;secondly,the weighted sum of CrossEntropy Loss and Dice loss is designed as the mask loss to improve the segmentation accuracy and robustness of the model;finally,the non-maximal suppression(NMS)algorithm to better handle the overlap problem in instance segmentation.Experimental results show that the proposed method achieves an average segmentation accuracy of mAP of 80.4% on three types of electrical equipment datasets,including transformers,insulators and voltage transformers,which improve the detection accuracy by more than 5.7% compared with the original Solov2 model.The segmentation model proposed can provide a focusing technical means for the intelligent management of power systems.

基于改进Q-learning算法的移动机器人路径规划

针对传统Q-learning算法应用在路径规划中存在收敛速度慢、运行时间长、学习效率差等问题,提出一种将人工势场法和传统Q-learning算法结合的改进Q-learning算法。该算法引入人工势场法的引力函数与斥力函数,通过对比引力函数动态选择奖励值,以及对比斥力函数计算姿值,动态更新Q值,使移动机器人具有目的性的探索,并且优先选择离障碍物较远的位置移动。通过仿真实验证明,与传统Q-learning算法、引入引力场算法对比,改进Q-learning算法加快了收敛速度,缩短了运行时间,提高了学习效率,降低了与障碍物相撞的概率,使移动机器人能够快速地找到一条无碰撞通路。

Weighted multilinear p-adic Hardy operators and commutators on p-adic Herz-type spaces

In this paper,we introduce the weighted multilinear p-adic Hardy operator and weighted multilinear p-adic Ces`aro operator,we also obtain the boundedness of these two operators on the product of p-adic Herz spaces and p-adic Morrey-Herz spaces,the corresponding operator norms are also established in each case.Moreover,the boundedness of commutators of these two operators with symbols in central bounded mean oscillation spaces and Lipschitz spaces on p-adic Morrey-Herz spaces are also given.