An Intelligent Framework for Resilience Recovery of FANETs with Spatio-Temporal Aggregation and Multi-Head Attention Mechanism

Due to the time-varying topology and possible disturbances in a conflict environment,it is still challenging to maintain the mission performance of flying Ad hoc networks(FANET),which limits the application of Unmanned Aerial Vehicle(UAV)swarms in harsh environments.This paper proposes an intelligent framework to quickly recover the cooperative coveragemission by aggregating the historical spatio-temporal network with the attention mechanism.The mission resilience metric is introduced in conjunction with connectivity and coverage status information to simplify the optimization model.A spatio-temporal node pooling method is proposed to ensure all node location features can be updated after destruction by capturing the temporal network structure.Combined with the corresponding Laplacian matrix as the hyperparameter,a recovery algorithm based on the multi-head attention graph network is designed to achieve rapid recovery.Simulation results showed that the proposed framework can facilitate rapid recovery of the connectivity and coverage more effectively compared to the existing studies.The results demonstrate that the average connectivity and coverage results is improved by 17.92%and 16.96%,respectively compared with the state-of-the-art model.Furthermore,by the ablation study,the contributions of each different improvement are compared.The proposed model can be used to support resilient network design for real-time mission execution.

Structured Multi-Head Attention Stock Index Prediction Method Based Adaptive Public Opinion Sentiment Vector

The present study examines the impact of short-term public opinion sentiment on the secondary market,with a focus on the potential for such sentiment to cause dramatic stock price fluctuations and increase investment risk.The quantification of investment sentiment indicators and the persistent analysis of their impact has been a complex and significant area of research.In this paper,a structured multi-head attention stock index prediction method based adaptive public opinion sentiment vector is proposed.The proposedmethod utilizes an innovative approach to transform numerous investor comments on social platforms over time into public opinion sentiment vectors expressing complex sentiments.It then analyzes the continuous impact of these vectors on the market through the use of aggregating techniques and public opinion data via a structured multi-head attention mechanism.The experimental results demonstrate that the public opinion sentiment vector can provide more comprehensive feedback on market sentiment than traditional sentiment polarity analysis.Furthermore,the multi-head attention mechanism is shown to improve prediction accuracy through attention convergence on each type of input information separately.Themean absolute percentage error(MAPE)of the proposedmethod is 0.463%,a reduction of 0.294% compared to the benchmark attention algorithm.Additionally,the market backtesting results indicate that the return was 24.560%,an improvement of 8.202% compared to the benchmark algorithm.These results suggest that themarket trading strategy based on thismethod has the potential to improve trading profits.

Posture Detection of Heart Disease Using Multi-Head Attention Vision Hybrid(MHAVH)Model

Cardiovascular disease is the leading cause of death globally.This disease causes loss of heart muscles and is also responsible for the death of heart cells,sometimes damaging their functionality.A person’s life may depend on receiving timely assistance as soon as possible.Thus,minimizing the death ratio can be achieved by early detection of heart attack(HA)symptoms.In the United States alone,an estimated 610,000 people die fromheart attacks each year,accounting for one in every four fatalities.However,by identifying and reporting heart attack symptoms early on,it is possible to reduce damage and save many lives significantly.Our objective is to devise an algorithm aimed at helping individuals,particularly elderly individuals living independently,to safeguard their lives.To address these challenges,we employ deep learning techniques.We have utilized a vision transformer(ViT)to address this problem.However,it has a significant overhead cost due to its memory consumption and computational complexity because of scaling dot-product attention.Also,since transformer performance typically relies on large-scale or adequate data,adapting ViT for smaller datasets is more challenging.In response,we propose a three-in-one steam model,theMulti-Head Attention Vision Hybrid(MHAVH).Thismodel integrates a real-time posture recognition framework to identify chest pain postures indicative of heart attacks using transfer learning techniques,such as ResNet-50 and VGG-16,renowned for their robust feature extraction capabilities.By incorporatingmultiple heads into the vision transformer to generate additional metrics and enhance heart-detection capabilities,we leverage a 2019 posture-based dataset comprising RGB images,a novel creation by the author that marks the first dataset tailored for posture-based heart attack detection.Given the limited online data availability,we segmented this dataset into gender categories(male and female)and conducted testing on both segmented and original datasets.The training accuracy of our model reached an impressive 99.77%.Upon testing,the accuracy for male and female datasets was recorded at 92.87%and 75.47%,respectively.The combined dataset accuracy is 93.96%,showcasing a commendable performance overall.Our proposed approach demonstrates versatility in accommodating small and large datasets,offering promising prospects for real-world applications.

Multi-Head Attention Spatial-Temporal Graph Neural Networks for Traffic Forecasting

Accurate traffic prediction is crucial for an intelligent traffic system (ITS). However, the excessive non-linearity and complexity of the spatial-temporal correlation in traffic flow severely limit the prediction accuracy of most existing models, which simply stack temporal and spatial modules and fail to capture spatial-temporal features effectively. To improve the prediction accuracy, a multi-head attention spatial-temporal graph neural network (MSTNet) is proposed in this paper. First, the traffic data is decomposed into unique time spans that conform to positive rules, and valuable traffic node attributes are mined through an adaptive graph structure. Second, time and spatial features are captured using a multi-head attention spatial-temporal module. Finally, a multi-step prediction module is used to achieve future traffic condition prediction. Numerical experiments were conducted on an open-source dataset, and the results demonstrate that MSTNet performs well in spatial-temporal feature extraction and achieves more positive forecasting results than the baseline methods.

Using Recurrent Neural Network Structure and Multi-Head Attention with Convolution for Fraudulent Phone Text Recognition

Fraud cases have been a risk in society and people’s property security has been greatly threatened.In recent studies,many promising algorithms have been developed for social media offensive text recognition as well as sentiment analysis.These algorithms are also suitable for fraudulent phone text recognition.Compared to these tasks,the semantics of fraudulent words are more complex and more difficult to distinguish.Recurrent Neural Networks(RNN),the variants ofRNN,ConvolutionalNeuralNetworks(CNN),and hybrid neural networks to extract text features are used by most text classification research.However,a single network or a simple network combination cannot obtain rich characteristic knowledge of fraudulent phone texts relatively.Therefore,a new model is proposed in this paper.In the fraudulent phone text,the knowledge that can be learned by the model includes the sequence structure of sentences,the correlation between words,the correlation of contextual semantics,the feature of keywords in sentences,etc.The new model combines a bidirectional Long-Short Term Memory Neural Network(BiLSTM)or a bidirectional Gate Recurrent United(BiGRU)and a Multi-Head attention mechanism module with convolution.A normalization layer is added after the output of the final hidden layer.BiLSTM or BiGRU is used to build the encoding and decoding layer.Multi-head attention mechanism module with convolution(MHAC)enhances the ability of the model to learn global interaction information and multi-granularity local interaction information in fraudulent sentences.A fraudulent phone text dataset is produced by us in this paper.The THUCNews data sets and fraudulent phone text data sets are used in experiments.Experiment results show that compared with the baseline model,the proposed model(LMHACL)has the best experiment results in terms of Accuracy,Precision,Recall,and F1 score on the two data sets.And the performance indexes on fraudulent phone text data sets are all above 0.94.

Discharge Summaries Based Sentiment Detection Using Multi-Head Attention and CNN-BiGRU

Automatic extraction of the patient’s health information from the unstructured data concerning the discharge summary remains challenging.Discharge summary related documents contain various aspects of the patient health condition to examine the quality of treatment and thereby help improve decision-making in the medical field.Using a sentiment dictionary and feature engineering,the researchers primarily mine semantic text features.However,choosing and designing features requires a lot of manpower.The proposed approach is an unsupervised deep learning model that learns a set of clusters embedded in the latent space.A composite model including Active Learning(AL),Convolutional Neural Network(CNN),BiGRU,and Multi-Attention,called ACBMA in this research,is designed to measure the quality of treatment based on discharge summaries text sentiment detection.CNN is utilized for extracting the set of local features of text vectors.Then BiGRU network was utilized to extract the text’s global features to solve the issues that a single CNN cannot obtain global semantic information and the traditional Recurrent Neural Network(RNN)gradient disappearance.Experiments prove that the ACBMA method can demonstrate the effectiveness of the suggested method,achieve comparable results to state-of-arts methods in sentiment detection,and outperform them with accurate benchmarks.Finally,several algorithm studies ultimately determined that the ACBMA method is more precise for discharge summaries sentiment analysis.

基于Multi-head Attention和Bi-LSTM的实体关系分类

关系分类是自然语言处理领域的一项重要任务,能够为知识图谱的构建、问答系统和信息检索等提供技术支持.与传统关系分类方法相比较,基于神经网络和注意力机制的关系分类模型在各种关系分类任务中都获得了更出色的表现.以往的模型大多采用单层注意力机制,特征表达相对单一.因此本文在已有研究基础上,引入多头注意力机制(Multi-head attention),旨在让模型从不同表示空间上获取关于句子更多层面的信息,提高模型的特征表达能力.同时在现有的词向量和位置向量作为网络输入的基础上,进一步引入依存句法特征和相对核心谓词依赖特征,其中依存句法特征包括当前词的依存关系值和所依赖的父节点位置,从而使模型进一步获取更多的文本句法信息.在SemEval-2010 任务8 数据集上的实验结果证明,该方法相较之前的深度学习模型,性能有进一步提高.

基于A-Capsule的多标签文本分类研究

目前在进行多标签文本分类任务中,大部分算法忽视了句子中词语的关键程度以及句子中词与词之间的相互联系。提出一种A-Capsule(Attention-Capsule,注意力胶囊网络)模型,由Capsule网络和Attention机制结合而得到。使文本数据转换成机器可以识别的向量化形式,然后使Multi-Head-Attention机制对不同单词进行学习,以此来确定词语的重要程度,利用Capsule网络的动态路由算法提取文本中的局部空间特征信息,使用分类器进行分类。实验使用的数据集来自今日头条下的数据集,使用其中的新闻标题多标签数据集进行多组对比实验,依据实验结果得出,提出的组合模型在分类任务中,有较好的性能。

Multi-head attention-based long short-term memory model for speech emotion recognition

To fully make use of information from different representation subspaces,a multi-head attention-based long short-term memory(LSTM)model is proposed in this study for speech emotion recognition(SER).The proposed model uses frame-level features and takes the temporal information of emotion speech as the input of the LSTM layer.Here,a multi-head time-dimension attention(MHTA)layer was employed to linearly project the output of the LSTM layer into different subspaces for the reduced-dimension context vectors.To provide relative vital information from other dimensions,the output of MHTA,the output of feature-dimension attention,and the last time-step output of LSTM were utilized to form multiple context vectors as the input of the fully connected layer.To improve the performance of multiple vectors,feature-dimension attention was employed for the all-time output of the first LSTM layer.The proposed model was evaluated on the eNTERFACE and GEMEP corpora,respectively.The results indicate that the proposed model outperforms LSTM by 14.6%and 10.5%for eNTERFACE and GEMEP,respectively,proving the effectiveness of the proposed model in SER tasks.

基于Capsule-BiGRU的文本相似度分析算法

针对传统神经网络模型不能很好地提取文本特征的问题,提出基于capsule-BiGRU的文本相似度分析方法,该方法将胶囊网络(capsule)提取的文本的局部特征矩阵和双向门控循环单元网络(Bi GRU)提取的文本的全局特征矩阵分别进行相似度分析,得到文本的相似度矩阵,将相似度矩阵融合,得到两个文本的多层次相似度向量,从而进行文本相似度的判定。将传统的胶囊网络进行改进,把与文本语义无关的单词视为噪声胶囊,赋予较小权值,从而减轻对后续任务的影响。针对文本相似度的任务,在文本特征矩阵提取前加入互注意力机制,对于待分析的两个文本,通过计算一个文本中单词与另一文本中所有单词的相似度来对词向量赋予权值,从而能更准确地判断文本的相似度。在Quora Questions Pairs数据集进行实验,实验结果表明所提出的方法准确率为86.16%,F1值为88.77%,结果优于其他方法。

Study on the fusion emotion classification of multiple characteristics based on attention mechanism

The current research on emotional classification uses many methods that combine the attention mechanism with neural networks.However,the effect is unsatisfactory when dealing with complex text.An emotional classification model is proposed,which combines multi-head attention(MHA)with improved structured-self attention(SSA).The model makes several different linear transformations of input by introducing MHA mechanism and can extract more comprehensive high-level phrase representation features from the word embedded vector.Meanwhile,it can realize the parallelization calculation and ensure the training speed of the model.The improved SSA structure uses matrices to represent different parts of a sentence to extract local key information,to ensure that the degree of dependence between words is not affected by time and sentence length,and generate the overall semantics of the sentence.Experiment results show that the current model effectively obtains global structural information and improves classification accuracy.

基于改进YOLOv5的胶囊内窥镜病灶区域检测

针对目前胶囊内窥镜病灶检测模型存在检测疾病单一且效率低等问题,提出了一种基于YOLOv5的胶囊内窥镜病灶区域检测方法。该方法在原始YOLOv5基础上进行了如下改进:首先,在主干网络Backbone部分,添加一个CBAM(convolutional block attention module)模块,增强模型对重要特征的突出能力;其次,在头部网络Head部分,添加一个检测头,增强模型对小目标的检测能力;最后,将原始YOLOv5的泛化交并比(generalized intersection over union, GIoU)损失函数替换成完整交并比(complete intersection over union, CIoU)损失函数,使模型训练时更快地收敛。本文提出的方法在长江大学第一临床医学院提供的胶囊内窥镜影像数据上进行了实验,精确率达到了93.6%,召回率达到了94.3%,mAP@0.5达到了97.2%,而且检测速度达到了每帧0.027 2 s。实验结果表明提出的方法是有效的、灵活的、鲁棒的,能够满足临床医学诊断的实际需求。

基于空洞分层注意力胶囊网络的X射线焊缝缺陷识别方法

由于X射线焊缝图像的复杂多样性,使得很多传统基于X射线焊缝缺陷检测方法的准确性不高,泛化能力较差。提出一种基于空洞分层注意力胶囊网络(DHACNet)的X射线焊缝缺陷识别方法。DHACNet由卷积模块、空洞分层注意力和胶囊网络(CapsNet)组成。卷积模块用来提取图像的卷积特征,空洞分层注意力用来提取多尺度显著性特征,CapsNet利用胶囊层和动态路由算法替代卷积神经网络(CNN)中的池化操作和全连接操作。DHACNet具有强大多尺度特征提取能力,能够克服CNN只关注图像局部特征和池化操作导致图像部分信息丢失等不足。在构建的X射线焊缝缺陷图像集上进行识别试验,识别准确率为96%以上,与传统方法进行比较,结果表明,该方法有效可行,能够为X射线焊缝缺陷识别系统提供技术支持。

基于胶囊异构图注意力网络的中文表格型数据事实验证

事实验证旨在检查一个文本陈述是否被给定的证据所支持.由于表格结构上具有依赖性、内容上具有隐含性,以表格作为证据的事实验证任务仍面临很多挑战.现有工作或者利用逻辑表达式来解析基于表格证据的陈述,或者设计表格感知神经网络来编码陈述-表格对,以此实现基于表格的事实验证任务.但是,这些方法没有充分利用陈述背后隐含的表格信息,从而导致模型的推理性能下降,并且基于表格证据的中文陈述具有更加复杂的语法和语义,也给模型推理带来更大的困难.为此,提出基于胶囊异构图注意力网络(CapsHAN)的中文表格型数据事实验证方法,所提方法能充分理解陈述的结构和语义,进而挖掘和利用陈述所隐含的表格信息,有效提升基于表格的事实验证任务准确性.具体而言,首先通过对陈述进行依存句法分析和命名实体识别来构建异构图,接着对该图采用异构图注意力网络和胶囊图神经网络进行学习和理解,然后将得到的陈述文本表示与经过编码的表格文本表示进行拼接,最后完成结果的预测.更进一步,针对现有中文表格型事实验证数据集匮乏而难以支持基于表格的事实验证方法性能评价的难题,首先对主流TABFACT和INFOTABS表格事实验证英文数据集进行中文转化,并且专门针对中文表格型数据的特点构建了基于UCL国家标准的数据集UCLDS,该数据集将维基百科信息框作为人工注释的自然语言陈述的证据,并被标记为蕴含、反驳或中立3类.UCLDS在同时支持单表和多表推理方面比传统TABFACT和INFOTABS数据集更胜一筹.在上述3个中文基准数据集上的实验结果表明,所提模型的表现均优于基线模型,证明该模型在基于中文表格的事实验证任务上的优越性.

融合注意力机制的胶囊网络方面级情感分析

方面级情感分析旨在明确文本中关于特定方面的情感极性。针对句中方面词由复杂词组组成造成方面情感极性判断错误的问题,论文提出了一种融合注意力机制的胶囊网络方面级情感分析模型。模型首先通过双向长短时记忆网络(Bi-directional Long Short-Term Memory,BiLSTM)提取序列语义信息,使用N-gram模型对序列语义信息中的目标方面进行编码,然后利用交互注意力机制学习方面词和上下文之间的注意力,将最终生成的文本表示接入融合方面特征表示的胶囊网络进行分类,得到文本方面级的情感分类结果。模型利用胶囊网络有效提取部分与整体关系特征的能力,融合N-gram模型提取到的方面特征变换矩阵,改进了传统动态路由方法,增强了模型对方面情感极性的判断能力。该模型在SemEval-2014餐馆数据集和笔记本数据集上与多个模型进行了对比分析,实验结果显示该模型在两个数据集上的精确度达到了78.4%和72.4%,F1分数分别为0.687和0.668,证明融合交互注意力机制的胶囊网络模型在方面级情感分析任务方面具有较强的分类效果。

多阶段注意力胶囊网络的图像分类

针对传统的胶囊网络(Capsule network,CapsNet)特征提取不充分的问题,提出一种图像分类的多阶段注意力胶囊网络模型.首先,在卷积层对低层特征和高层特征分别采用注意力(Spatial attention,SA)和通道注意力(Channel attention,CA)来提取有效特征;然后,提出基于向量的注意力(Vector attention,VA)机制作用于动态路由层,增加对重要胶囊的关注,进而提高低层胶囊对高层胶囊预测的准确性;最后,在五个公共数据集上进行图像分类的对比实验.结果表明,所提出的CapsNet模型在分类精度和鲁棒性上优于其他胶囊网络模型,在仿射变换图像重构方面也表现良好.

基于双分支多头注意力的场景图生成方法

针对已有场景图生成模型获取上下文信息有限的问题,提出一种有效的上下文融合模块,即双分支多头注意力(dual-stream multi-head attention, DMA)模块,并将DMA分别用于物体分类阶段和关系分类阶段,基于此提出基于双分支多头注意力的场景图生成网络(dual-stream multi-head attention-based scene graph generation network, DMA-Net)。该网络由目标检测、物体语义解析和关系语义解析3个模块组成。首先,通过目标检测模块定位图像中的物体并提取物体特征;其次,使用物体语义解析模块中的节点双分支多头注意力(object dual-stream multi-head attention, O-DMA)获取融合了节点上下文的特征,该特征经过物体语义解码器获得物体类别标签;最后,通过关系语义解析模块中的边双分支多头注意力(relationship dual-stream multi-head attention, R-DMA)输出融合了边上下文的特征,该特征经过关系语义解码器输出关系类别标签。在公开的视觉基因组(visual genome, VG)数据集上分别计算了DMA-Net针对场景图检测、场景图分类和谓词分类3个子任务的图约束召回率和无图约束召回率,并与主流的场景图生成方法进行比较。实验结果表明,所提出的方法能够充分挖掘场景中的上下文信息,基于上下文增强的特征表示有效提升了场景图生成任务的精度。

基于多脑区注意力机制胶囊融合网络的EEG-fNIRS情感识别

为了提高情感识别的准确率,提出多脑区注意力机制和胶囊融合模块的胶囊网络模型(MBA-CFc CapsNet).通过情感视频片段诱发采集EEG-fNIRS信号,构建TYUT3.0数据集.提取EEG和f NIRS的特征,将其映射到矩阵,通过多脑区注意力机制融合EEG和fNIRS的特征,给予不同脑区特征不同的权重,以提取质量更高的初级胶囊.使用胶囊融合模块,减少进入动态路由机制的胶囊数量,减少模型运行的时间.利用MBA-CFc CapsNet模型在TYUT3.0情感数据集上进行实验,与单模态EEG和f NIRS识别结果相比,2种信号结合情感识别的准确率提高了1.53%和14.35%.MBA-CF-cCapsNet模型与原始CapsNet模型相比,平均识别率提高了4.98%,与当前常用的CapsNet情感识别模型相比提高了1%~5%.

基于CapsNet和SRU的工业互联网入侵检测方法

随着工业互联网的普及,工业系统中大量的基础设施和设备接入互联网,使得工业系统更容易受到外部攻击,工业互联网入侵检测成为保障工业网络安全和稳定运营的重要手段。针对现有的深度学习方法在工业互联网入侵检测中存在数据特征提取不全和对罕见攻击检测准确率低的问题,提出一种基于胶囊网络(CapsNet)和简单循环单元(SRU)融合的工业互联网入侵检测模型。采用SMOTE-ENN算法完成数据的平衡处理,结合简单循环单元神经网络和引入残差块的胶囊网络分别提取流量数据的时间和空间特征,通过自注意力机制进行特征加权,进而提高模型的检测性能。在气体管道数据集上,对比经过SMOTE-ENN算法处理前后的预测结果,该模型对MSCI和MFCI类别的识别精度分别提高4.69百分点和4.41百分点,表明数据平衡算法提高了分类器对少数类样本的预测能力,对比其他模型,该模型的准确率达到99.36%,误报率为0.73%。

基于应变数据和改进MHA模型的囊体缺陷检测

针对浮空器囊体表面缺陷难以检测等问题,提出一种基于应变时序数据和改进多头注意力(multi-head attention,简称MHA)模型的囊体缺陷检测方法,该方法通过对囊体声波激励后的应变时序数据进行端到端特征提取与检测,实现浮空器囊体缺陷检测。首先,在声波激励下,通过粘贴在囊体表面的应变片收集同位置无裂纹和有裂纹时的应变时序数据;其次,将收集的应变时序数据按照一定的长度划分样本,每个样本划分为多个时序向量的组合并输入到改进MHA模型中,提取隐藏于时序数据中的缺陷特征;然后,网络输出各个时序样本相应的缺陷识别结果;最后,在收集到的囊体应变数据上将该方法与其他4种传统模型的检测结果进行对比。结果表明,该方法平均检测准确率为97.7%,优于其他4种模型,验证了该方法的有效性。