An Intelligent Framework for Resilience Recovery of FANETs with Spatio-Temporal Aggregation and Multi-Head Attention Mechanism

Due to the time-varying topology and possible disturbances in a conflict environment,it is still challenging to maintain the mission performance of flying Ad hoc networks(FANET),which limits the application of Unmanned Aerial Vehicle(UAV)swarms in harsh environments.This paper proposes an intelligent framework to quickly recover the cooperative coveragemission by aggregating the historical spatio-temporal network with the attention mechanism.The mission resilience metric is introduced in conjunction with connectivity and coverage status information to simplify the optimization model.A spatio-temporal node pooling method is proposed to ensure all node location features can be updated after destruction by capturing the temporal network structure.Combined with the corresponding Laplacian matrix as the hyperparameter,a recovery algorithm based on the multi-head attention graph network is designed to achieve rapid recovery.Simulation results showed that the proposed framework can facilitate rapid recovery of the connectivity and coverage more effectively compared to the existing studies.The results demonstrate that the average connectivity and coverage results is improved by 17.92%and 16.96%,respectively compared with the state-of-the-art model.Furthermore,by the ablation study,the contributions of each different improvement are compared.The proposed model can be used to support resilient network design for real-time mission execution.

Structured Multi-Head Attention Stock Index Prediction Method Based Adaptive Public Opinion Sentiment Vector

The present study examines the impact of short-term public opinion sentiment on the secondary market,with a focus on the potential for such sentiment to cause dramatic stock price fluctuations and increase investment risk.The quantification of investment sentiment indicators and the persistent analysis of their impact has been a complex and significant area of research.In this paper,a structured multi-head attention stock index prediction method based adaptive public opinion sentiment vector is proposed.The proposedmethod utilizes an innovative approach to transform numerous investor comments on social platforms over time into public opinion sentiment vectors expressing complex sentiments.It then analyzes the continuous impact of these vectors on the market through the use of aggregating techniques and public opinion data via a structured multi-head attention mechanism.The experimental results demonstrate that the public opinion sentiment vector can provide more comprehensive feedback on market sentiment than traditional sentiment polarity analysis.Furthermore,the multi-head attention mechanism is shown to improve prediction accuracy through attention convergence on each type of input information separately.Themean absolute percentage error(MAPE)of the proposedmethod is 0.463%,a reduction of 0.294% compared to the benchmark attention algorithm.Additionally,the market backtesting results indicate that the return was 24.560%,an improvement of 8.202% compared to the benchmark algorithm.These results suggest that themarket trading strategy based on thismethod has the potential to improve trading profits.

Posture Detection of Heart Disease Using Multi-Head Attention Vision Hybrid(MHAVH)Model

Cardiovascular disease is the leading cause of death globally.This disease causes loss of heart muscles and is also responsible for the death of heart cells,sometimes damaging their functionality.A person’s life may depend on receiving timely assistance as soon as possible.Thus,minimizing the death ratio can be achieved by early detection of heart attack(HA)symptoms.In the United States alone,an estimated 610,000 people die fromheart attacks each year,accounting for one in every four fatalities.However,by identifying and reporting heart attack symptoms early on,it is possible to reduce damage and save many lives significantly.Our objective is to devise an algorithm aimed at helping individuals,particularly elderly individuals living independently,to safeguard their lives.To address these challenges,we employ deep learning techniques.We have utilized a vision transformer(ViT)to address this problem.However,it has a significant overhead cost due to its memory consumption and computational complexity because of scaling dot-product attention.Also,since transformer performance typically relies on large-scale or adequate data,adapting ViT for smaller datasets is more challenging.In response,we propose a three-in-one steam model,theMulti-Head Attention Vision Hybrid(MHAVH).Thismodel integrates a real-time posture recognition framework to identify chest pain postures indicative of heart attacks using transfer learning techniques,such as ResNet-50 and VGG-16,renowned for their robust feature extraction capabilities.By incorporatingmultiple heads into the vision transformer to generate additional metrics and enhance heart-detection capabilities,we leverage a 2019 posture-based dataset comprising RGB images,a novel creation by the author that marks the first dataset tailored for posture-based heart attack detection.Given the limited online data availability,we segmented this dataset into gender categories(male and female)and conducted testing on both segmented and original datasets.The training accuracy of our model reached an impressive 99.77%.Upon testing,the accuracy for male and female datasets was recorded at 92.87%and 75.47%,respectively.The combined dataset accuracy is 93.96%,showcasing a commendable performance overall.Our proposed approach demonstrates versatility in accommodating small and large datasets,offering promising prospects for real-world applications.

Multi-Head Attention Spatial-Temporal Graph Neural Networks for Traffic Forecasting

Accurate traffic prediction is crucial for an intelligent traffic system (ITS). However, the excessive non-linearity and complexity of the spatial-temporal correlation in traffic flow severely limit the prediction accuracy of most existing models, which simply stack temporal and spatial modules and fail to capture spatial-temporal features effectively. To improve the prediction accuracy, a multi-head attention spatial-temporal graph neural network (MSTNet) is proposed in this paper. First, the traffic data is decomposed into unique time spans that conform to positive rules, and valuable traffic node attributes are mined through an adaptive graph structure. Second, time and spatial features are captured using a multi-head attention spatial-temporal module. Finally, a multi-step prediction module is used to achieve future traffic condition prediction. Numerical experiments were conducted on an open-source dataset, and the results demonstrate that MSTNet performs well in spatial-temporal feature extraction and achieves more positive forecasting results than the baseline methods.

Using Recurrent Neural Network Structure and Multi-Head Attention with Convolution for Fraudulent Phone Text Recognition

Fraud cases have been a risk in society and people’s property security has been greatly threatened.In recent studies,many promising algorithms have been developed for social media offensive text recognition as well as sentiment analysis.These algorithms are also suitable for fraudulent phone text recognition.Compared to these tasks,the semantics of fraudulent words are more complex and more difficult to distinguish.Recurrent Neural Networks(RNN),the variants ofRNN,ConvolutionalNeuralNetworks(CNN),and hybrid neural networks to extract text features are used by most text classification research.However,a single network or a simple network combination cannot obtain rich characteristic knowledge of fraudulent phone texts relatively.Therefore,a new model is proposed in this paper.In the fraudulent phone text,the knowledge that can be learned by the model includes the sequence structure of sentences,the correlation between words,the correlation of contextual semantics,the feature of keywords in sentences,etc.The new model combines a bidirectional Long-Short Term Memory Neural Network(BiLSTM)or a bidirectional Gate Recurrent United(BiGRU)and a Multi-Head attention mechanism module with convolution.A normalization layer is added after the output of the final hidden layer.BiLSTM or BiGRU is used to build the encoding and decoding layer.Multi-head attention mechanism module with convolution(MHAC)enhances the ability of the model to learn global interaction information and multi-granularity local interaction information in fraudulent sentences.A fraudulent phone text dataset is produced by us in this paper.The THUCNews data sets and fraudulent phone text data sets are used in experiments.Experiment results show that compared with the baseline model,the proposed model(LMHACL)has the best experiment results in terms of Accuracy,Precision,Recall,and F1 score on the two data sets.And the performance indexes on fraudulent phone text data sets are all above 0.94.

Discharge Summaries Based Sentiment Detection Using Multi-Head Attention and CNN-BiGRU

Automatic extraction of the patient’s health information from the unstructured data concerning the discharge summary remains challenging.Discharge summary related documents contain various aspects of the patient health condition to examine the quality of treatment and thereby help improve decision-making in the medical field.Using a sentiment dictionary and feature engineering,the researchers primarily mine semantic text features.However,choosing and designing features requires a lot of manpower.The proposed approach is an unsupervised deep learning model that learns a set of clusters embedded in the latent space.A composite model including Active Learning(AL),Convolutional Neural Network(CNN),BiGRU,and Multi-Attention,called ACBMA in this research,is designed to measure the quality of treatment based on discharge summaries text sentiment detection.CNN is utilized for extracting the set of local features of text vectors.Then BiGRU network was utilized to extract the text’s global features to solve the issues that a single CNN cannot obtain global semantic information and the traditional Recurrent Neural Network(RNN)gradient disappearance.Experiments prove that the ACBMA method can demonstrate the effectiveness of the suggested method,achieve comparable results to state-of-arts methods in sentiment detection,and outperform them with accurate benchmarks.Finally,several algorithm studies ultimately determined that the ACBMA method is more precise for discharge summaries sentiment analysis.

Narrow Pooling Clothing Classification Based on Attention Mechanism

In recent years,with the rapid development of e-commerce,people need to classify the wide variety and a large number of clothing images appearing on e-commerce platforms.In order to solve the problems of long time consumption and unsatisfactory classification accuracy arising from the classification of a large number of clothing images,researchers have begun to exploit deep learning techniques instead of traditional learning methods.The paper explores the use of convolutional neural networks(CNNs)for feature learning to enhance global feature information interactions by adding an improved hybrid attention mechanism(HAM)that fully utilizes feature weights in three dimensions:channel,height,and width.Moreover,the improved pooling layer not only captures local feature information,but also fuses global and local information to improve the misclassification problem that occurs between similar categories.Experiments on the Fashion-MNIST and DeepFashion datasets show that the proposed method significantly improves the accuracy of clothing classification(93.62%and 67.9%)compared with residual network(ResNet)and convolutional block attention module(CBAM).

基于Multi-head Attention和Bi-LSTM的实体关系分类

关系分类是自然语言处理领域的一项重要任务,能够为知识图谱的构建、问答系统和信息检索等提供技术支持.与传统关系分类方法相比较,基于神经网络和注意力机制的关系分类模型在各种关系分类任务中都获得了更出色的表现.以往的模型大多采用单层注意力机制,特征表达相对单一.因此本文在已有研究基础上,引入多头注意力机制(Multi-head attention),旨在让模型从不同表示空间上获取关于句子更多层面的信息,提高模型的特征表达能力.同时在现有的词向量和位置向量作为网络输入的基础上,进一步引入依存句法特征和相对核心谓词依赖特征,其中依存句法特征包括当前词的依存关系值和所依赖的父节点位置,从而使模型进一步获取更多的文本句法信息.在SemEval-2010 任务8 数据集上的实验结果证明,该方法相较之前的深度学习模型,性能有进一步提高.

Multi-head attention-based long short-term memory model for speech emotion recognition

To fully make use of information from different representation subspaces,a multi-head attention-based long short-term memory(LSTM)model is proposed in this study for speech emotion recognition(SER).The proposed model uses frame-level features and takes the temporal information of emotion speech as the input of the LSTM layer.Here,a multi-head time-dimension attention(MHTA)layer was employed to linearly project the output of the LSTM layer into different subspaces for the reduced-dimension context vectors.To provide relative vital information from other dimensions,the output of MHTA,the output of feature-dimension attention,and the last time-step output of LSTM were utilized to form multiple context vectors as the input of the fully connected layer.To improve the performance of multiple vectors,feature-dimension attention was employed for the all-time output of the first LSTM layer.The proposed model was evaluated on the eNTERFACE and GEMEP corpora,respectively.The results indicate that the proposed model outperforms LSTM by 14.6%and 10.5%for eNTERFACE and GEMEP,respectively,proving the effectiveness of the proposed model in SER tasks.

基于BiLSTM-Attention-CNN混合神经网络的文本分类方法

BiLSTM和CNN网络在文本分类领域取得了较好的应用效果,二者的结合可以充分发挥CNN的特征提取能力和BiLSTM的上下文依赖能力,但是没有体现出每个词语在文本中的重要程度,无法将注意力集中在重要的词上。针对该问题,提出一种基于BiLSTM-Attention-CNN混合神经网络的文本分类方法。在BiLSTM层之后加入注意力机制(Attention)提取输出信息的注意力分值;注意力层之后,连接k-max池化层,提取前k个重要的词,增强模型特征的表达能力。在DBPedia和AGNews数据集上进行实验,结果表明,该模型相较于其他现有网络模型,分类准确率提高1~2个百分点。

基于AT_CNN与Attention-BiGRU融合网络的电网故障报修信息的自动分类研究

随着经济和信息化的快速发展,电网规模的不断扩大,电网用户的故障报修数量呈逐年上升的趋势。针对目前电网用户故障报修工单难以自动化分析和处理的问题,提出一种基于AT_CNN算法与Attention-BiGRU的融合网络,该网络可将以上两种互补模型提取的局部特征和整体特征进行拼接融合。其中的AT_CNN算法利用Attention池化与Top k池化结合的方法对池化层进行改进,能更好提取上下文的文本特征。结果表明,该方法在公共数据集上的分类准确率较传统深度学习方法显著提高,在电网故障报修工单数据上的分类准确率可以达到95.71%。

Study on the fusion emotion classification of multiple characteristics based on attention mechanism

The current research on emotional classification uses many methods that combine the attention mechanism with neural networks.However,the effect is unsatisfactory when dealing with complex text.An emotional classification model is proposed,which combines multi-head attention(MHA)with improved structured-self attention(SSA).The model makes several different linear transformations of input by introducing MHA mechanism and can extract more comprehensive high-level phrase representation features from the word embedded vector.Meanwhile,it can realize the parallelization calculation and ensure the training speed of the model.The improved SSA structure uses matrices to represent different parts of a sentence to extract local key information,to ensure that the degree of dependence between words is not affected by time and sentence length,and generate the overall semantics of the sentence.Experiment results show that the current model effectively obtains global structural information and improves classification accuracy.

Attention Neural Network for User Behavior Modeling

The recommendation system can effectively and quickly provide valuable information for users by filtering out massive useless data. User behavior modeling can extract all kinds of aggregated features over the heterogeneous behaviors to help recommendation. However, the existing user behavior modeling method cannot solve the cold-start problem caused by data sparse. Recent recommender systems which exploit reviews for learning representation can alleviate the above problem to a certain extent. Therefore, a user behavior modeling is proposed for recommendation task using attention neural network based on user reviews (AT-UBM). Firstly vanilla attention was used to sample reviews, and then CNN+Pooling method was applied to extract user behavior features. Finally the long-term behavior was combined with short-term behavior in feature spaces. Experimental results on real datasets show that the review-based user behavior model has better prediction accuracy and generalization capability.

基于自适应融合的实时车辆检测

针对传统的车辆检测技术检测速度慢和精度低的问题,提出了一种融合注意力的自适应金字塔网络的交通目标检测算法(fusion attentiont adaptive pyramid network,FAAP-Net),可以显著降低交通事故的发生率。为了降低计算复杂度,设计了一种轻量级的互补池化结构(CPS),该结构在宽度和高度上采用了两组不同的池化组合,在保持高精度的同时,显著降低了网络的浮点运算数(GFLOPs)和参数量。为了解决智能交通系统特征图生成过程中的信息损失问题,通过将自适应注意力模块(AAM)和特征增强模块(FEM)引入自适应融合特征金字塔网络(AF-FPN),以融入车辆检测的形状特征。针对车辆细节特征表征弱的问题,引入了一种按通道维度分组的注意力(SA)机制,以增强主干网络对不同车辆检测细节特征的关注,有效提取车辆细节的显著特征。在BDD100K数据集上的实验结果表明,FAAP-Net算法相比于传统算法,平均精度从30.3%提升到43.7%。

基于改进YOLOv8的煤矿输送带异物检测

现有基于深度学习的输送带异物检测模型较大,难以在边缘设备部署,且对不同尺寸异物和小目标异物存在错检、漏检情况。针对上述问题,提出一种基于改进YOLOv8的煤矿输送带异物检测方法。采用深度可分离卷积、压缩和激励(SE)网络将YOLOv8主干网络中C2f模块的Bottleneck重新构建为DSBlock,在保持模型轻量化的同时提升检测性能;为增强对不同尺寸目标物体信息的获取能力,引入高效通道注意力(ECA)机制,并对ECA的输入层进行自适应平均池化和自适应最大池化操作,得到跨通道交互MECA模块,以增强模块的全局视觉信息,进一步提升异物识别精度;将YOLOv8的3个检测头修改为4个轻量化小目标检测头,以增强对小目标的敏感性,有效降低小目标异物的漏检率和错检率。实验结果表明:改进YOLOv8的精确度达91.69%,mAP@50达92.27%,较YOLOv8分别提升了3.09%和4.07%;改进YOLOv8的检测速度达73.92帧/s,可充分满足煤矿输送带异物实时检测的需求;改进YOLOv8的精确度、mAP@50、参数量、权重大小和每秒浮点运算数均优于SSD,Faster-RCNN,YOLOv5,YOLOv7-tiny等主流目标检测算法。

融合注意力和扩张卷积的遥感影像道路信息提取方法

针对高分辨率遥感影像语义分割存在地物边缘分割不连续、道路及背景特征复杂多样导致道路提取分割精度不高的问题,提出了一种融合双通道注意力和扩张卷积的遥感影像道路信息提取语义分割网络(A 2DU-Net)。首先,在特征提取部分引入坐标注意力(coordinate attention,CA)模块,捕捉道路位置、方向和跨通道信息,精确定位道路信息。其次,针对网络对细节特征丢失的敏感问题,在编码器的末端利用不同扩张率的空洞卷积构建多尺度特征融合的空洞空间金字塔池化模块(multi-scale Atrous spatial pyramid pooling module,MASPPM)来获得更大的感受野,提高网络性能。最后,为了避免U-Net中纯跳跃连接在语义上不相似特征的融合,在编码器和解码器的跳跃连接之间增加了双通道注意力机制来实现门控筛选,抑制非目标区域的特征,提高网络的分割精度。实验在公共道路数据集Massachusetts上对网络模型进行测试,OA(准确率)、交并比(IoU)、平均交并比(mIoU)和F1等评价指标分别达到98.07%、64.39%、81.20%和88.67%。与主流方法U-Net和DDUNet进行比较,mIoU分别提升了3.07%、0.22%,IoU分别提升了1.98%、0.52%。实验结果表明,所提出的方法优于所有的比较方法,能够有效提高道路分割的精确度。

基于改进图注意力网络的油井产量预测模型

针对图注意力网络处理噪声和时序数据较弱,并且在堆叠多层后出现梯度爆炸、过平滑等问题,提出一种改进图注意力网络模型.首先,使用Squeeze-and-Excitation模块对样本输入数据的特征信息进行不同程度关注,增强模型处理噪声的能力;其次,使用多头注意力机制,将序列数据中每个序列相对其他序列进行加权求和,提取数据的时序性;再次,将图注意力网络提取的节点特征与节点的度中心性拼接,获取节点的局部特征,并用全局平均池化的方式提取节点的全局特征;最后,将两者进行融合得到节点的最终特征表示,增强模型的表征能力.为验证改进图注意力网络的有效性,将改进图注意力网络模型与LSTM,GRU和GGNN模型进行对比,实验结果表明,该模型预测效果得到有效提升,具有更高的预测精度.

基于改进YOLOv5s的道路裂缝检测算法

为了解决道路巡检系统光学传感器采集的裂缝图像中颜色特征不明显且尺寸不规则造成检测精度不高、泛化能力不足的问题,提出改进YOLOv5s的裂缝检测算法。将结合深度可分离卷积(Depthwise Separable Convolution, DSC)的全局注意力(Global Attention Mechanism, GAM)引入主干特征提取网络,在降低注意力复杂度的同时获得丰富的跨维度特征,增强了裂缝的识别能力;采用空间金字塔软池化网络(Spatial Pyramid Softpool, SPSF),通过Softpool池化保留多维语义以减少信息弥散,提高了边界框回归的准确性;在颈部特征增强网络,运用空洞深度可分离卷积(Atrous DSC)进行下采样,通过扩大感受野加强深层和浅层信息的聚合能力,提高裂缝识别的泛化性。经过在自制道路裂缝数据集上的实验,相较于YOLOv5s,改进算法的mAP提高2.2%,有效提升了道路裂缝检测的准确性和对不同背景下裂缝识别的泛化能力。

融合Inception V1-CBAM-CNN的轴承剩余寿命预测模型

针对现有的滚动轴承剩余寿命(RUL)预测方法精度低、轴承健康指标(HI)构建困难等问题,提出了一种基于卷积神经网络(CNN)并融合Inception V1模块和卷积注意力机制模块(CBAM)的滚动轴承RUL预测模型。首先,在CNN中添加了CBAM机制,并进行了加权处理,在通道和空间维度对重要特征进行了强化,对次要特征进行了抑制,通过添加改进的InceptionV1模块,提高了CNN通道间信息交互水平,全面提取了退化特征;然后,进行了网络优化,采用全局最大池化(GMP)方法对模型进行了简化,采用Dropout和批量归一化(BN)方法,避免了过拟合,提高了精度,且克服了训练时出现的梯度消失问题;最后,对数据进行了处理,将降噪后的信号重组为三维张量,将其作为HI,构建了退化标签,引入了评价指标,采用PHM2012轴承数据集进行了实验验证,在3种工况下将其与深度神经网络(DNN)、CNN方法、结合注意力机制的残差网络方法(ResNet)进行了对比。研究结果表明:该方法在变负载条件下的平均RMSE为0.033,较其他方法的RMSE值分别降低了86%、78%和69%,在预测精度和泛化能力方面具有明显优势。

基于改进SegNet的鸡只检测算法

为实现智能化检测出鸡场中死亡鸡只,提出一种基于改进语义分割模型AT-SegNet的鸡只检测算法。基于对称编码解码结构SegNet,利用空洞卷积在解码前聚合不同感受野的上下文信息,设计一种三尺度注意力级联融合模块,以并联方式嵌入编、解码器间,丰富解码器信息。利用多层深度可分离卷积替代标准卷积,提取深层次语义信息,减少计算量提高实时性。将鸡群图像分割结果交并比与阈值对比判别鸡只状态。实验结果表明,改进的AT-SegNet较原算法的检测精度提高了25.17%,能够在复杂鸡群环境中准确、高效地发现死亡鸡只。