Constrained Multi-Objective Optimization With Deep Reinforcement Learning Assisted Operator Selection

Solving constrained multi-objective optimization problems with evolutionary algorithms has attracted considerable attention.Various constrained multi-objective optimization evolutionary algorithms(CMOEAs)have been developed with the use of different algorithmic strategies,evolutionary operators,and constraint-handling techniques.The performance of CMOEAs may be heavily dependent on the operators used,however,it is usually difficult to select suitable operators for the problem at hand.Hence,improving operator selection is promising and necessary for CMOEAs.This work proposes an online operator selection framework assisted by Deep Reinforcement Learning.The dynamics of the population,including convergence,diversity,and feasibility,are regarded as the state;the candidate operators are considered as actions;and the improvement of the population state is treated as the reward.By using a Q-network to learn a policy to estimate the Q-values of all actions,the proposed approach can adaptively select an operator that maximizes the improvement of the population according to the current state and thereby improve the algorithmic performance.The framework is embedded into four popular CMOEAs and assessed on 42 benchmark problems.The experimental results reveal that the proposed Deep Reinforcement Learning-assisted operator selection significantly improves the performance of these CMOEAs and the resulting algorithm obtains better versatility compared to nine state-of-the-art CMOEAs.

Enhancing Hyper-Spectral Image Classification with Reinforcement Learning and Advanced Multi-Objective Binary Grey Wolf Optimization

Hyperspectral(HS)image classification plays a crucial role in numerous areas including remote sensing(RS),agriculture,and the monitoring of the environment.Optimal band selection in HS images is crucial for improving the efficiency and accuracy of image classification.This process involves selecting the most informative spectral bands,which leads to a reduction in data volume.Focusing on these key bands also enhances the accuracy of classification algorithms,as redundant or irrelevant bands,which can introduce noise and lower model performance,are excluded.In this paper,we propose an approach for HS image classification using deep Q learning(DQL)and a novel multi-objective binary grey wolf optimizer(MOBGWO).We investigate the MOBGWO for optimal band selection to further enhance the accuracy of HS image classification.In the suggested MOBGWO,a new sigmoid function is introduced as a transfer function to modify the wolves’position.The primary objective of this classification is to reduce the number of bands while maximizing classification accuracy.To evaluate the effectiveness of our approach,we conducted experiments on publicly available HS image datasets,including Pavia University,Washington Mall,and Indian Pines datasets.We compared the performance of our proposed method with several state-of-the-art deep learning(DL)and machine learning(ML)algorithms,including long short-term memory(LSTM),deep neural network(DNN),recurrent neural network(RNN),support vector machine(SVM),and random forest(RF).Our experimental results demonstrate that the Hybrid MOBGWO-DQL significantly improves classification accuracy compared to traditional optimization and DL techniques.MOBGWO-DQL shows greater accuracy in classifying most categories in both datasets used.For the Indian Pine dataset,the MOBGWO-DQL architecture achieved a kappa coefficient(KC)of 97.68%and an overall accuracy(OA)of 94.32%.This was accompanied by the lowest root mean square error(RMSE)of 0.94,indicating very precise predictions with minimal error.In the case of the Pavia University dataset,the MOBGWO-DQL model demonstrated outstanding performance with the highest KC of 98.72%and an impressive OA of 96.01%.It also recorded the lowest RMSE at 0.63,reinforcing its accuracy in predictions.The results clearly demonstrate that the proposed MOBGWO-DQL architecture not only reaches a highly accurate model more quickly but also maintains superior performance throughout the training process.

Feature Selection with Deep Reinforcement Learning for Intrusion Detection System

An intrusion detection system(IDS)becomes an important tool for ensuring security in the network.In recent times,machine learning(ML)and deep learning(DL)models can be applied for the identification of intrusions over the network effectively.To resolve the security issues,this paper presents a new Binary Butterfly Optimization algorithm based on Feature Selection with DRL technique,called BBOFS-DRL for intrusion detection.The proposed BBOFSDRL model mainly accomplishes the recognition of intrusions in the network.To attain this,the BBOFS-DRL model initially designs the BBOFS algorithm based on the traditional butterfly optimization algorithm(BOA)to elect feature subsets.Besides,DRL model is employed for the proper identification and classification of intrusions that exist in the network.Furthermore,beetle antenna search(BAS)technique is applied to tune the DRL parameters for enhanced intrusion detection efficiency.For ensuring the superior intrusion detection outcomes of the BBOFS-DRL model,a wide-ranging experimental analysis is performed against benchmark dataset.The simulation results reported the supremacy of the BBOFS-DRL model over its recent state of art approaches.

A Heterogeneous Information Fusion Deep Reinforcement Learning for Intelligent Frequency Selection of HF Communication

The high-frequency(HF) communication is one of essential communication methods for military and emergency application. However, the selection of communication frequency channel is always a difficult problem as the crowded spectrum, the time-varying channels, and the malicious intelligent jamming. The existing frequency hopping, automatic link establishment and some new anti-jamming technologies can not completely solve the above problems. In this article, we adopt deep reinforcement learning to solve this intractable challenge. First, the combination of the spectrum state and the channel gain state is defined as the complex environmental state, and the Markov characteristic of defined state is analyzed and proved. Then, considering that the spectrum state and channel gain state are heterogeneous information, a new deep Q network(DQN) framework is designed, which contains multiple sub-networks to process different kinds of information. Finally, aiming to improve the learning speed and efficiency, the optimization targets of corresponding sub-networks are reasonably designed, and a heterogeneous information fusion deep reinforcement learning(HIF-DRL) algorithm is designed for the specific frequency selection. Simulation results show that the proposed algorithm performs well in channel prediction, jamming avoidance and frequency channel selection.

Multi-Agent Few-Shot Meta Reinforcement Learning for Trajectory Design and Channel Selection in UAV-Assisted Networks

Unmanned aerial vehicle(UAV)-assisted communications have been considered as a solution of aerial networking in future wireless networks due to its low-cost, high-mobility, and swift features. This paper considers a UAV-assisted downlink transmission,where UAVs are deployed as aerial base stations to serve ground users. To maximize the average transmission rate among the ground users, this paper formulates a joint optimization problem of UAV trajectory design and channel selection, which is NP-hard and non-convex. To solve the problem, we propose a multi-agent deep Q-network(MADQN) scheme.Specifically, the agents that the UAVs act as perform actions from their observations distributively and share the same reward. To tackle the tasks where the experience is insufficient, we propose a multi-agent meta reinforcement learning algorithm to fast adapt to the new tasks. By pretraining the tasks with similar distribution, the learning model can acquire general knowledge. Simulation results have indicated the MADQN scheme can achieve higher throughput than fixed allocation. Furthermore, our proposed multiagent meta reinforcement learning algorithm learns the new tasks much faster compared with the MADQN scheme.

Enhancing cut selection through reinforcement learning

With the rapid development of artificial intelligence in recent years,applying various learning techniques to solve mixed-integer linear programming(MILP)problems has emerged as a burgeoning research domain.Apart from constructing end-to-end models directly,integrating learning approaches with some modules in the traditional methods for solving MILPs is also a promising direction.The cutting plane method is one of the fundamental algorithms used in modern MILP solvers,and the selection of appropriate cuts from the candidate cuts subset is crucial for enhancing efficiency.Due to the reliance on expert knowledge and problem-specific heuristics,classical cut selection methods are not always transferable and often limit the scalability and generalizability of the cutting plane method.To provide a more efficient and generalizable strategy,we propose a reinforcement learning(RL)framework to enhance cut selection in the solving process of MILPs.Firstly,we design feature vectors to incorporate the inherent properties of MILP and computational information from the solver and represent MILP instances as bipartite graphs.Secondly,we choose the weighted metrics to approximate the proximity of feasible solutions to the convex hull and utilize the learning method to determine the weights assigned to each metric.Thirdly,a graph convolutional neural network is adopted with a self-attention mechanism to predict the value of weighting factors.Finally,we transform the cut selection process into a Markov decision process and utilize RL method to train the model.Extensive experiments are conducted based on a leading open-source MILP solver SCIP.Results on both general and specific datasets validate the effectiveness and efficiency of our proposed approach.

Joint Topology Construction and Power Adjustment for UAV Networks:A Deep Reinforcement Learning Based Approach

In this paper,we investigate a backhaul framework jointly considering topology construction and power adjustment for self-organizing UAV networks.To enhance the backhaul rate with limited information exchange and avoid malicious power competition,we propose a deep reinforcement learning(DRL)based method to construct the backhaul framework where each UAV distributedly makes decisions.First,we decompose the backhaul framework into three submodules,i.e.,transmission target selection(TS),total power control(PC),and multi-channel power allocation(PA).Then,the three submodules are solved by heterogeneous DRL algorithms with tailored rewards to regulate UAVs’behaviors.In particular,TS is solved by deep-Q learning to construct topology with less relay and guarantee the backhaul rate.PC and PA are solved by deep deterministic policy gradient to match the traffic requirement with proper finegrained transmission power.As a result,the malicious power competition is alleviated,and the backhaul rate is further enhanced.Simulation results show that the proposed framework effectively achieves system-level and all-around performance gain compared with DQL and max-min method,i.e.,higher backhaul rate,lower transmission power,and fewer hop.

A Reinforcement Learning System for Fault Detection and Diagnosis in Mechatronic Systems

With the increasing demand for the automation of operations and processes in mechatronic systems,fault detection and diagnosis has become a major topic to guarantee the process performance.There exist numerous studies on the topic of applying artificial intelligence methods for fault detection and diagnosis.However,much of the focus has been given on the detection of faults.In terms of the diagnosis of faults,on one hand,assumptions are required,which restricts the diagnosis range.On the other hand,different faults with similar symptoms cannot be distinguished,especially when the model is not trained by plenty of data.In this work,we proposed a reinforcement learning system for fault detection and diagnosis.No assumption is required.Feature exaction is first made.Then with the features as the states of the environment,the agent directly interacts with the environment.Optimal policy,which determines the exact category,size and location of the fault,is obtained by updating Q values.The method takes advantage of expert knowledge.When the features are unclear,action will be made to get more information from the new state for further determination.We create recurrent neural network with the long short-term memory architecture to approximate Q values.The application on a motor is discussed.The experimental results validate that the proposed method demonstrates a significant improvement compared with existing state-of-the-art methods of fault detection and diagnosis.

Material Selection of a Natural Fibre Reinforced Polymer Composites using an Analytical Approach

Material selection has become a critical part of design for engineers,due to availability of diverse choice of materials that have similar properties and meet the product design specification.Implementation of statistical analysis alone makes it difficult to identify the ideal composition of the final composite.An integrated approach between statistical model and micromechanical model is desired.In this paper,resultant natural fibre and polymer matrix from previous study is used to estimate the mechanical properties such as density,Young’s modulus and tensile strength.Four levels of fibre loading are used to compare the optimum natural fibre reinforced polymer composite(NFRPC).The result from this analytical approach revealed that kenaf/polystyrene(PS)with 40%fibre loading is the ideal composite in automotive component application.It was found that the ideal composite score is 1.156 g/cm^(3),24.2 GPa and 413.4 MPa for density,Young’s modulus and tensile strength,respectively.A suggestion to increase the properties on Young’s modulus are also presented.This work proves that the statistical model is well incorporated with the analytical approach to choose the correct composite to use in automotive application.

基于强化学习的大规模多模Mesh网络联合路由选择及资源调度算法

为了平衡新型电力系统中大规模多模Mesh网络的传输可靠性和效率,该文在对优化问题进行描述和分析的基础上提出一种基于强化学习的大规模多模Mesh网络联合路由选择及资源调度算法,分为两个阶段。在第1阶段中,根据网络拓扑结构信息和业务需求,利用一种多条最短路径路由算法,输出所有最短路径。在第2阶段中,提出一种基于多臂老虎机(MAB)的资源调度算法,该算法基于得到的最短路径集合构建MAB的摇臂,然后根据业务需求计算回报,最终给出最优的路由选择及资源调度方式用于业务传输。仿真结果表明,所提算法能够满足不同的业务传输需求,实现端到端路径的平均时延和平均传输成功率的高效平衡。

气肿疽梭菌C54-1株的制备与检定

为研究气肿疽梭菌C54-1株(CVCC60001)的菌种特性,制备了1批气肿疽梭菌C54-1株,并对菌种的形态及生化特性、培养特性、血清学特性、真空度、纯粹、剩余水分、毒力及免疫原性、16S rDNA等进行检定,以及对于该菌的适宜培养基促生长能力等方面进行了比较。实验结果表明,该冻干菌种的形态及生化特性、培养特性、血清学特性、真空度、纯粹、剩余水分、毒力及免疫原性均符合《中华人民共和国兽用生物制品规程》二〇〇〇版质量标准的规定。16S rDNA鉴定为气肿疽梭菌,相似度为99.93%。在免疫原性检定中,使用致死剂量攻毒时,免疫组能够4/4被保护,证明该菌明矾苗免疫效果良好。使用商品化梭菌培养基对于该菌培养进行了比对,证实商品化培养基替代厌气肉肝汤的可能性。本研究为气肿疽梭菌的制备和检定提供参考依据,并使用不同培养基进行比对研究,为该菌的稳定培养和疫苗工艺改进提供了研究基础。

基于路径搜索DQN的特殊车辆路线优化策略

为保障特殊车辆在复杂且易拥堵的城市交通环境下执行紧急任务的时效性与畅通性,提出一种基于路径搜索式深度Q网络(P-DQN)的特殊车辆路线优化策略。采用回溯法协助深度Q网络(DQN)解决路径搜索过程中的死路、回路问题,利用人工势场机制引导DQN搜索路径,避免路径结果过长。结合轮盘赌选择法与贪婪值自适应调整机制进一步提升DQN选取路段和建议行驶速度时的准确性。实验在InTAS数据集上对真实城市交通进行模拟,与RERoute、CH等SOTA方法相比,P-DQN获得的总价值提高约16%。

对人工接地装置材料选择要求的探讨

依据电化学腐蚀的原理对人工接地装置材质选择的要求进行研究,着重分析以镀锌扁钢为代表的不同材质的人工接地埋地敷设时的腐蚀过程,并针对不同的工程情况提出相应的解决方案。

基于深度强化学习的无蜂窝系统无线接入点选择算法

面向以用户为中心的无蜂窝分布式多输入多输出(Multiple-Input Multiple-Output, MIMO)架构,研究利用不完备信道状态信息(Channel State Information, CSI)下实现无线接入点(Access Point, AP)与用户(User Equipment, UE)之间的选择,提出基于深度强化学习(Deep Reinforcement Learning, DRL)的高效分配算法,通过使用不完备CSI快速生成以用户为中心的AP集合,减少了对前馈链路容量的占用。仿真结果表明,与其他传统选择算法相比,所提出的DRL接入点选择算法可以获得至少22.48%的总遍历频谱效率增益;与深度Q网络(Deep-Q-Network, DQN)算法相比,可以获得约14.17%的总频谱效率增益。

动态决策驱动的工控网络数据要素威胁检测方法

近年来,工控网络发展势头迅猛.其数字化、智能化、自动化的优势为工业带来巨大效益的同时,也面临着愈发复杂多变的攻击威胁.在数据要素安全的背景下,及时发现和应对工控网络威胁成为一项迫切需要得到解决的任务.通过对工控网络中的数据流进行连续监测和分析,工控网络威胁检测问题可以转化为时间序列异常检测问题.然而现有时间序列异常检测方法受限于工控网络数据集的质量,且往往仅对单一类型异常敏感而忽略其他异常.针对上述问题,提出了一种基于深度强化学习和数据增强的工控网络威胁检测方法(deep reinforcement learning and data augmentation based threat detection method in industrial control networks,DELTA).该方法提出了一种新的时序数据集数据增强选择方法,可以针对不同的基准模型选择合适的数据增强操作集以提升工控网络时间序列数据集的质量;同时使用深度强化学习算法(A2C/PPO)在不同时间点从基线模型中动态选取候选模型,以利用多种类型的异常检测模型解决单一类型异常敏感问题.与现有时间序列异常检测模型对比的实验结果表明,在付出可接受的额外时间消耗成本下,DELTA在准确率和F1值上比所有基线模型有明显的提升,验证了方法的有效性与实用性.

水深点与等深线协同综合的强化学习方法

针对当前海图中水深点与等深线两要素的自动综合过程相对独立,二者相互影响考虑不够充分,导致结果不够理想的问题,提出一种水深点与等深线协同综合的强化学习方法。首先,获取用于水深点与等深线协同综合的训练样本;然后,构建并训练强化学习模型,挖掘水深点与等深线在综合过程中的相互影响关系;最后,利用训练后的模型,动态自适应调整水深点与等深线的综合策略。试验结果表明,在航海安全保证的可靠性、制图综合结果的图理性、海底地貌表达的准确性及海图要素分布的美观性等方面,强化学习法的性能均要优于当前常见的简单综合法、冲突避免法及协调水深法,更适用于处理水深点与等深线的协同综合问题。

多期贝叶斯强化学习鲁棒投资组合选择模型

在传统多期分布式鲁棒投资组合选择模型中,不确定集合的估计是一个具有挑战性的难题。使用贝叶斯强化学习方法来动态更新不确定集合中的一、二阶矩等模型参数,进而研究贝叶斯强化学习框架下均值–最坏鲁棒CVaR模型的求解问题。通过结合动态规划和渐进对冲算法,设计了两层分解求解框架。下层通过求解一系列二阶锥规划来得到给定模型参数下子问题的最优策略,上层使用贝叶斯公式得到可实施的非预期投资策略。基于美国股票市场的实证结果表明:多期鲁棒强化学习投资组合选择模型相较传统模型具有更好的样本外投资表现。

基于强化学习的混合元启发式暂态电压稳定特征选择方法及可解释性研究

新型电力系统发展背景下,使用有效的特征选择方法来提取与暂态电压稳定强相关的关键响应特征,对研究暂态电压失稳机理与系统潜在安全隐患具有重要意义。为此,提出一种基于改进过滤法与混合元启发式包装法的复合框架进行特征选择的新方法。基于对称不确定性值改进的最大相关最小冗余性准则进行特征粗筛;将Q学习强化学习融合至元启发式优化算法中,并采用开发探索折衷策略以增强特征细选能力,获取最优关键响应特征子集。在此基础上,采用沙普利值加性解释归因理论综合分析各筛选特征对暂态电压稳定的影响与系统薄弱环节。新型电力系统算例验证了所提方法的有效性。

信息年龄敏感的稀疏移动群智感知机制

稀疏移动群智感知是一种新兴的模式,它从感知区域的子集收集数据,然后推理其他区域的数据.然而,在实际应用中,工人不足或分布不均的情况广泛存在.因此,在有限的预算下,必须优先选择相对更重要的工人收集数据.此外,许多稀疏移动群智感知应用对数据的时效性要求较高.因此本文将考虑数据的新鲜度,并使用信息年龄作为新鲜度指标.为了解决这些挑战,本文提出了一种轻量级年龄敏感的数据感知和推理框架.该框架旨在预算约束下,选择合适的工人收集数据,并通过准确捕捉感知数据时空关系进行数据推理,以优化信息年龄和推理的准确性.由于预算和工人有限,可能会导致数据量较少的情况.因此,本文还提出了精简数据推理模型的方法,以提高推理效率.通过广泛的实验进一步论证了该框架在实际应用中的优越性.

超长超重地连墙钢筋笼吊装技术研究

超长超重钢筋笼吊装作业是地连墙施工中危险性较大的环节,文章通过力学理论分析计算,确定了钢筋笼吊装的加固措施及主、副吊起重机的选型,研究了超长超重地连墙钢筋笼吊装过程中各吊点的最不利工况,采用钢筋笼分节空中回直、槽口拼接方法,解决了超长超重钢筋笼整体吊装难、安全风险高、设备选型大的难题,提高了吊装作业的安全性,以供参考。