改进Q-Learning的路径规划算法研究

针对Q-Learning算法学习效率低、收敛速度慢且在动态障碍物的环境下路径规划效果不佳的问题,本文提出一种改进Q-Learning的移动机器人路径规划算法.针对该问题,算法根据概率的突变性引入探索因子来平衡探索和利用以加快学习效率;通过在更新函数中设计深度学习因子以保证算法探索概率;融合遗传算法,避免陷入局部路径最优同时按阶段探索最优迭代步长次数,以减少动态地图探索重复率;最后提取输出的最优路径关键节点采用贝塞尔曲线进行平滑处理,进一步保证路径平滑度和可行性.实验通过栅格法构建地图,对比实验结果表明,改进后的算法效率相较于传统算法在迭代次数和路径上均有较大优化,且能够较好的实现动态地图下的路径规划,进一步验证所提方法的有效性和实用性.

改进的Q-learning蜂群算法求解置换流水车间调度问题

针对置换流水车间调度问题,提出了一种基于改进的Q-learning算法的人工蜂群算法。该算法设计了一种改进的奖励函数作为人工蜂群算法的环境,根据奖励函数的优劣来判断下一代种群的寻优策略,并通过Q-learning智能选择人工蜂群算法的蜜源的更新维度数大小,根据选择的维度数大小对编码进行更新,提高了收敛速度和精度,最后使用不同规模的置换流水车间调度问题的实例来验证所提算法的性能,通过对标准实例的计算与其它算法对比,证明该算法的准确性。

基于改进Q-Learning的移动机器人路径规划算法

随着移动机器人在生产生活中的深入应用,其路径规划能力也需要向快速性和环境适应性兼备发展。为解决现有移动机器人使用强化学习方法进行路径规划时存在的探索前期容易陷入局部最优、反复搜索同一区域,探索后期收敛率低、收敛速度慢的问题,本研究提出一种改进的Q-Learning算法。该算法改进Q矩阵赋值方法,使迭代前期探索过程具有指向性,并降低碰撞的情况;改进Q矩阵迭代方法,使Q矩阵更新具有前瞻性,避免在一个小区域中反复探索;改进随机探索策略,在迭代前期全面利用环境信息,后期向目标点靠近。在不同栅格地图仿真验证结果表明,本文算法在Q-Learning算法的基础上,通过上述改进降低探索过程中的路径长度、减少抖动并提高收敛的速度,具有更高的计算效率。

基于Q-Learning的航空器滑行路径规划研究

针对传统算法规划航空器滑行路径准确度低、不能根据整体场面运行情况进行路径规划的问题,提出一种基于Q-Learning的路径规划方法。通过对机场飞行区网络结构模型和强化学习的仿真环境分析,设置了状态空间和动作空间,并根据路径的合规性和合理性设定了奖励函数,将路径合理性评价值设置为滑行路径长度与飞行区平均滑行时间乘积的倒数。最后,分析了动作选择策略参数对路径规划模型的影响。结果表明,与A*算法和Floyd算法相比,基于Q-Learning的路径规划在滑行距离最短的同时,避开了相对繁忙的区域,路径合理性评价值高。

基于Q-Learning的分簇无线传感网信任管理机制

针对无线传感器网络中存在的安全问题,提出了基于Q-Learning的分簇无线传感网信任管理机制(Q-learning based trust management mechanism for clustered wireless sensor networks,QLTMM-CWSN).该机制主要考虑通信信任、数据信任和能量信任3个方面.在网络运行过程中,基于节点的通信行为、数据分布和能量消耗,使用Q-Learning算法更新节点信任值,并选择簇内信任值最高的节点作为可信簇头节点.当簇中主簇头节点的信任值低于阈值时,可信簇头节点代替主簇头节点管理簇内成员节点,维护正常的数据传输.研究结果表明,QLTMM-CWSN机制能有效抵御通信攻击、伪造本地数据攻击、能量攻击和混合攻击.

基于多步信息辅助的Q-learning路径规划算法

为提升静态环境下移动机器人路径规划能力,解决传统Q-learning算法在路径规划中收敛速度慢的问题,提出一种基于多步信息辅助机制的Q-learning改进算法。利用ε-greedy策略中贪婪动作的多步信息与历史最优路径长度更新资格迹,使有效的资格迹在算法迭代中持续发挥作用,用保存的多步信息解决可能落入的循环陷阱;使用局部多花朵的花授粉算法初始化Q值表,提升机器人前期搜索效率;基于机器人不同探索阶段的目的,结合迭代路径长度的标准差与机器人成功到达目标点的次数设计动作选择策略,以增强算法对环境信息探索与利用的平衡能力。实验结果表明:该算法具有较快的收敛速度,验证了算法的可行性与有效性。

基于softmax的加权Double Q-Learning算法

强化学习作为机器学习的一个分支,用于描述和解决智能体在与环境的交互过程中,通过学习策略以达成回报最大化的问题。Q-Learning作为无模型强化学习的经典方法,存在过估计引起的最大化偏差问题,并且在环境中奖励存在噪声时表现不佳。Double Q-Learning(DQL)的出现解决了过估计问题,但同时造成了低估问题。为解决以上算法的高低估问题,提出了基于softmax的加权Q-Learning算法,并将其与DQL相结合,提出了一种新的基于softmax的加权Double Q-Learning算法(WDQL-Softmax)。该算法基于加权双估计器的构造,对样本期望值进行softmax操作得到权重,使用权重估计动作价值,有效平衡对动作价值的高估和低估问题,使估计值更加接近理论值。实验结果表明,在离散动作空间中,相比于Q-Learning算法、DQL算法和WDQL算法,WDQL-Softmax算法的收敛速度更快且估计值与理论值的误差更小。

Facile synthesis of hierarchical NaX zeolite from natural kaolinite for efficient Knoevenagel condensation

Zeolite catalysts have found extensive applications in the synthesis of various fine chemicals.However,the micropores of zeolites impose diffusion limitations on bulky molecules,greatly reducing the catalytic efficiency.Herein,we explore an economic and environmentally friendly method for synthesizing hierarchical NaX zeolite that exhibits improved catalytic performance in the Knoevenagel condensation reaction for producing the useful fine chemical 2-cyano-3-phenylacrylate.The synthesis was achieved via a low-temperature activation of kaolinite and subsequent in-situ transformation strategy without any template or seed.Systematic characterizations reveal that the synthesized NaX zeolite has both intercrystalline and intra-crystalline mesopores,smaller crystal size,and larger external specific surface area compared to commercial NaX zeolite.Detailed mechanism investigations show that the inter-crystalline mesopores are generated by stacking smaller crystals formed from in-situ crystallization of the depolymerized kaolinite,and the intra-crystalline mesopores are inherited from the pores in the depolymerized kaolinite.This synthesis strategy provides an energy-saving and effective way to construct hierarchical zeolites,which may gain wide applications in fine chemical manufacturing.

High-Performance All-Printed Flexible Micro-Supercapacitors with Hierarchical Encapsulation

Printed micro-supercapacitors(MSCs)have shown broad prospect in flexible and wearable electronics.Most of previous studies focused on printing the electrochemically active materials paying less attention to other key components like current collectors and electrolytes.This study presents an allprinting strategy to fabricate in-plane flexible and substrate-free MSCs with hierarchical encapsulation.This new type of“all-in-one”MSC is constructed by encapsulating the in-plane interdigital current collectors and electrodes within the polyvinyl-alcohol-based hydrogel electrolyte via sequential printing.The bottom electrolyte layer of this fully printed MSCs helps protect the device from the limitation of conventional substrate,showing excellent flexibility.The MSCs maintain a high capacitance retention of 96.84%even in a completely folded state.An optimal electrochemical performance can be achieved by providing ample and shorter transport paths for ions.The MSCs using commercial activated carbon as the active material are endowed with a high specific areal capacitance of 1892.90 mF cm^(-2)at a current density of 0.3 mA cm^(-2),and an outstanding volumetric energy density of 9.20 mWh cm^(-3)at a volumetric power density of 6.89 mW cm^(-3).For demonstration,a thermo-hygrometer is stably powered by five MSCs which are connected in series and wrapped onto a glass rod.This low-cost and versatile all-printing strategy is believed to diversify the application fields of MSCs with high capacitance and excellent flexibility.

Structural Engineering of Hierarchical Magnetic/Carbon Nanocomposites via In Situ Growth for High-Efficient Electromagnetic Wave Absorption

Materials exhibiting high-performance electromagnetic wave absorption have garnered considerable scientific and technological attention,yet encounter significant challenges.Developing new materials and innovative structural design concepts is crucial for expanding the application field of electromagnetic wave absorption.Particularly,hierarchical structure engineering has emerged as a promising approach to enhance the physical and chemical properties of materials,providing immense potential for creating versatile electromagnetic wave absorption materials.Herein,an exceptional multi-dimensional hierarchical structure was meticulously devised,unleashing the full microwave attenuation capabilities through in situ growth,selfreduction,and multi-heterogeneous interface integration.The hierarchical structure features a three-dimensional carbon framework,where magnetic nanoparticles grow in situ on the carbon skeleton,creating a necklace-like structure.Furthermore,magnetic nanosheets assemble within this framework.Enhanced impedance matching was achieved by precisely adjusting component proportions,and intelligent integration of diverse interfaces bolstered dielectric polarization.The obtain Fe_(3)O_(4)-Fe nanoparticles/carbon nanofibers/Al-Fe_(3)O_(4)-Fe nanosheets composites demonstrated outstanding performance with a minimum reflection loss(RLmin)value of−59.3 dB and an effective absorption bandwidth(RL≤−10 dB)extending up to 5.6 GHz at 2.2 mm.These notable accomplishments offer fresh insights into the precision design of high-efficient electromagnetic wave absorption materials.

多无人机辅助边缘计算场景下基于Q-learning的任务卸载优化

引入多无人机辅助边缘计算系统,由多个无人机和原有边缘服务器共同为移动用户提供通信和计算资源;将优化问题建模为资源竞争和卸载决策约束下的系统总效用最大化问题,系统总效用由用户满意度、任务延迟和系统能耗3个因素组成.由于优化模型是一个具有NP难属性的非凸问题,故采用强化学习方法求解得到使系统总效用最大的最优任务卸载决策集.仿真实验结果表明,与贪心顺序调优卸载方案和随机选择卸载方案相比,该文提出的Q-learning方案的系统总效用分别提高了15%和43%以上.

Lignite-Based Hierarchical Porous C/SiO_(x)Composites as High-Performance Anode for Potassium-Ion Batteries

Silicon oxide(SiO_(x),0<x≤2)has been recognized as a prominent anode material in lithium-ion batteries and sodium-ion batteries due to its high theoretical capacity,suitable electrochemical potential,and earth abundance.However,it is intrinsically poor electronic conductivity and excessive volume expansion during potassiation/depotassiation process hinder its application in potassium-ion batteries.Herein,we reported a hierarchical porous C/SiO_(x)potassium-ion batteries anode using lignite as raw material via a one-step carbonization and activation method.The amorphous C skeleton around SiO_(x)particles can effectively buffer the volume expansion,and improve the ionic/electronic conductivity and structural integrity,achieving outstanding rate capability and cyclability.As expected,the obtained C/SiO_(x)composite delivers a superb specific capacity of 370 mAh g^(-1)at 0.1 A g^(-1)after 100 cycles as well as a highly reversible capacity of 208 mAh g^(-1)after 1200 cycles at 1.0 A g^(-1).Moreover,the potassium ion storage mechanism of C/SiO_(x)electrodes was investigated by ex-situ X-ray diffraction and transmission electron microscopy,revealing the formation of reversible products of K_(6.8)Si_(45.3)and K_(4)SiO_(4),accompanied by generation of irreversible K2O after the first cycle.This work sheds light on designing low-cost Si-based anode materials for high-performance potassium-ion batteries and beyond.

基于改进Q-learning算法移动机器人局部路径研究

针对局部路径规划时因无法提前获取环境信息导致移动机器人搜索不到合适的路径,以及在采用马尔可夫决策过程中传统强化学习算法应用于局部路径规划时存在着学习效率低下及收敛速度较慢等问题,提出一种改进的Q-learn-ing(QL)算法。首先设计一种动态自适应贪婪策略,用于平衡移动机器人对环境探索和利用之间的问题;其次根据A*算法思想设计启发式学习评估模型,从而动态调整学习因子并为搜索路径提供导向作用;最后引入三阶贝塞尔曲线规划对路径进行平滑处理。通过Pycharm平台仿真结果表明,使得改进后的QL算法所规划的路径长度、搜索效率及路径平滑性等特性上都优于传统Sarsa算法及QL算法,比传统Sarsa算法迭代次数提高32.3%,搜索时间缩短27.08%,比传统QL算法迭代次数提高27.32%,搜索时间缩短17.28%,路径规划的拐点大幅度减少,局部路径优化效果较为明显。

一种基于Q-learning强化学习的导向性处理器安全性模糊测试方案

针对处理器安全性模糊测试在进行细粒度变异时遗传算法存在一定的盲目性,易使生成的测试用例触发相同类型漏洞的问题,提出了一种基于Q-learning强化学习的导向性处理器安全性模糊测试方案。通过测试用例的状态值和所触发的漏洞类型对应的权值构造奖励函数,使用强化学习指导生成具有针对性和导向性的测试用例,快速地触发不同类型的漏洞。在Hikey970平台上的实验验证了基于ARMv8的测试用例生成框架的有效性,并且相较于传统使用遗传算法作为反馈的策略,本文方案在相同时间内生成有效测试用例的的数量多19.15%,发现漏洞类型的数量多80.00%。

Q-Learning-Assisted Meta-Heuristics for Scheduling Distributed Hybrid Flow Shop Problems

The flow shop scheduling problem is important for the manufacturing industry.Effective flow shop scheduling can bring great benefits to the industry.However,there are few types of research on Distributed Hybrid Flow Shop Problems(DHFSP)by learning assisted meta-heuristics.This work addresses a DHFSP with minimizing the maximum completion time(Makespan).First,a mathematical model is developed for the concerned DHFSP.Second,four Q-learning-assisted meta-heuristics,e.g.,genetic algorithm(GA),artificial bee colony algorithm(ABC),particle swarm optimization(PSO),and differential evolution(DE),are proposed.According to the nature of DHFSP,six local search operations are designed for finding high-quality solutions in local space.Instead of randomselection,Q-learning assists meta-heuristics in choosing the appropriate local search operations during iterations.Finally,based on 60 cases,comprehensive numerical experiments are conducted to assess the effectiveness of the proposed algorithms.The experimental results and discussions prove that using Q-learning to select appropriate local search operations is more effective than the random strategy.To verify the competitiveness of the Q-learning assistedmeta-heuristics,they are compared with the improved iterated greedy algorithm(IIG),which is also for solving DHFSP.The Friedman test is executed on the results by five algorithms.It is concluded that the performance of four Q-learning-assisted meta-heuristics are better than IIG,and the Q-learning-assisted PSO shows the best competitiveness.

Traffic Control Based on Integrated Kalman Filtering and Adaptive Quantized Q-Learning Framework for Internet of Vehicles

Intelligent traffic control requires accurate estimation of the road states and incorporation of adaptive or dynamically adjusted intelligent algorithms for making the decision.In this article,these issues are handled by proposing a novel framework for traffic control using vehicular communications and Internet of Things data.The framework integrates Kalman filtering and Q-learning.Unlike smoothing Kalman filtering,our data fusion Kalman filter incorporates a process-aware model which makes it superior in terms of the prediction error.Unlike traditional Q-learning,our Q-learning algorithm enables adaptive state quantization by changing the threshold of separating low traffic from high traffic on the road according to the maximum number of vehicles in the junction roads.For evaluation,the model has been simulated on a single intersection consisting of four roads:east,west,north,and south.A comparison of the developed adaptive quantized Q-learning(AQQL)framework with state-of-the-art and greedy approaches shows the superiority of AQQL with an improvement percentage in terms of the released number of vehicles of AQQL is 5%over the greedy approach and 340%over the state-of-the-art approach.Hence,AQQL provides an effective traffic control that can be applied in today’s intelligent traffic system.

Kinetic-Thermodynamic Promotion Engineering toward High-Density Hierarchical and Zn-Doping Activity-Enhancing ZnNiO@CF for High-Capacity Desalination

Despite the promising potential of transition metal oxides(TMOs)as capacitive deionization(CDI)electrodes,the actual capacity of TMOs electrodes for sodium storage is significantly lower than the theoretical capacity,posing a major obstacle.Herein,we prepared the kinetically favorable Zn_(x)Ni_(1−x)O electrode in situ growth on carbon felt(Zn_(x)Ni_(1−x)O@CF)through constraining the rate of OH^(−)generation in the hydrothermal method.Zn_(x)Ni_(1−x)O@CF exhibited a high-density hierarchical nanosheet structure with three-dimensional open pores,benefitting the ion transport/electron transfer.And tuning the moderate amount of redox-inert Zn-doping can enhance surface electroactive sites,actual activity of redox-active Ni species,and lower adsorption energy,promoting the adsorption kinetic and thermodynamic of the Zn_(0.2)Ni_(0.8)O@CF.Benefitting from the kinetic-thermodynamic facilitation mechanism,Zn_(0.2)Ni_(0.8)O@CF achieved ultrahigh desalination capacity(128.9 mgNaCl g^(-1)),ultra-low energy consumption(0.164 kW h kgNaCl^(-1)),high salt removal rate(1.21 mgNaCl g^(-1) min^(-1)),and good cyclability.The thermodynamic facilitation and Na^(+)intercalation mechanism of Zn_(0.2)Ni_(0.8)O@CF are identified by the density functional theory calculations and electrochemical quartz crystal microbalance with dissipation monitoring,respectively.This research provides new insights into controlling electrochemically favorable morphology and demonstrates that Zn-doping,which is redox-inert,is essential for enhancing the electrochemical performance of CDI electrodes.

Deep Structure Optimization for Incremental Hierarchical Fuzzy Systems Using Improved Differential Evolution Algorithm

The optimization of the rule base of a fuzzy logic system (FLS) based on evolutionary algorithm has achievednotable results. However, due to the diversity of the deep structure in the hierarchical fuzzy system (HFS) and thecorrelation of each sub fuzzy system, the uncertainty of the HFS’s deep structure increases. For the HFS, a largenumber of studies mainly use fixed structures, which cannot be selected automatically. To solve this problem, thispaper proposes a novel approach for constructing the incremental HFS. During system design, the deep structureand the rule base of the HFS are encoded separately. Subsequently, the deep structure is adaptively mutated basedon the fitness value, so as to realize the diversity of deep structures while ensuring reasonable competition amongthe structures. Finally, the differential evolution (DE) is used to optimize the deep structure of HFS and theparameters of antecedent and consequent simultaneously. The simulation results confirm the effectiveness of themodel. Specifically, the root mean square errors in the Laser dataset and Friedman dataset are 0.0395 and 0.0725,respectively with rule counts of rules is 8 and 12, respectively.When compared to alternative methods, the resultsindicate that the proposed method offers improvements in accuracy and rule counts.

Hierarchical structures on platinum-iridium substrates enhancing conducting polymer adhesion

Conducting polymers(CPs),including poly(3,4-ethylenedioxythiophene):poly(styrenesulfonate)(PEDOT:PSS),are promising coating materials for neural electrodes.However,the weak adhesion of CP coatings to substrates such as platinum-iridium is a significant challenge that limits their practical application.To address this issue,we used femtosecond laser-prepared hierarchical structures on platinum-iridium(Pt-Ir)substrates to enhance the adhesion of PEDOT:PSS coatings.Next,we used cyclic voltammetry(CV)stress and accelerated aging tests to evaluate the stability of both drop cast and electrodeposited PEDOT:PSS coatings on Pt-Ir substrates,both with and without hierarchical structures.Our results showed that after 2000 CV cycles or five weeks of aging at 60℃,the morphology and electrochemical properties of the coatings on the Pt-Ir substrates with hierarchical structures remained relatively stable.In contrast,we found that smooth Pt-Ir substrate surfaces caused delamination of the PEDOT:PSS coating and exhibited both decreased charge storage capacity and increased impedance.Overall,enhancing the stability of PEDOT:PSS coatings used on common platinum-iridium neural electrodes offers great potential for improving their electrochemical performance and developing new functionalities.

The Immense Impact of Reverse Edges on Large Hierarchical Networks

Hierarchical networks are frequently encountered in animal groups,gene networks,and artificial engineering systems such as multiple robots,unmanned vehicle systems,smart grids,wind farm networks,and so forth.The structure of a large directed hierarchical network is often strongly influenced by reverse edges from lower-to higher-level nodes,such as lagging birds’howl in a flock or the opinions of lowerlevel individuals feeding back to higher-level ones in a social group.This study reveals that,for most large-scale real hierarchical networks,the majority of the reverse edges do not affect the synchronization process of the entire network;the synchronization process is influenced only by a small part of these reverse edges along specific paths.More surprisingly,a single effective reverse edge can slow down the synchronization of a huge hierarchical network by over 60%.The effect of such edges depends not on the network size but only on the average in-degree of the involved subnetwork.The overwhelming majority of active reverse edges turn out to have some kind of“bunching”effect on the information flows of hierarchical networks,which slows down synchronization processes.This finding refines the current understanding of the role of reverse edges in many natural,social,and engineering hierarchical networks,which might be beneficial for precisely tuning the synchronization rhythms of these networks.Our study also proposes an effective way to attack a hierarchical network by adding a malicious reverse edge to it and provides some guidance for protecting a network by screening out the specific small proportion of vulnerable nodes.