An Intelligent Framework for Resilience Recovery of FANETs with Spatio-Temporal Aggregation and Multi-Head Attention Mechanism

Due to the time-varying topology and possible disturbances in a conflict environment,it is still challenging to maintain the mission performance of flying Ad hoc networks(FANET),which limits the application of Unmanned Aerial Vehicle(UAV)swarms in harsh environments.This paper proposes an intelligent framework to quickly recover the cooperative coveragemission by aggregating the historical spatio-temporal network with the attention mechanism.The mission resilience metric is introduced in conjunction with connectivity and coverage status information to simplify the optimization model.A spatio-temporal node pooling method is proposed to ensure all node location features can be updated after destruction by capturing the temporal network structure.Combined with the corresponding Laplacian matrix as the hyperparameter,a recovery algorithm based on the multi-head attention graph network is designed to achieve rapid recovery.Simulation results showed that the proposed framework can facilitate rapid recovery of the connectivity and coverage more effectively compared to the existing studies.The results demonstrate that the average connectivity and coverage results is improved by 17.92%and 16.96%,respectively compared with the state-of-the-art model.Furthermore,by the ablation study,the contributions of each different improvement are compared.The proposed model can be used to support resilient network design for real-time mission execution.

Structured Multi-Head Attention Stock Index Prediction Method Based Adaptive Public Opinion Sentiment Vector

The present study examines the impact of short-term public opinion sentiment on the secondary market,with a focus on the potential for such sentiment to cause dramatic stock price fluctuations and increase investment risk.The quantification of investment sentiment indicators and the persistent analysis of their impact has been a complex and significant area of research.In this paper,a structured multi-head attention stock index prediction method based adaptive public opinion sentiment vector is proposed.The proposedmethod utilizes an innovative approach to transform numerous investor comments on social platforms over time into public opinion sentiment vectors expressing complex sentiments.It then analyzes the continuous impact of these vectors on the market through the use of aggregating techniques and public opinion data via a structured multi-head attention mechanism.The experimental results demonstrate that the public opinion sentiment vector can provide more comprehensive feedback on market sentiment than traditional sentiment polarity analysis.Furthermore,the multi-head attention mechanism is shown to improve prediction accuracy through attention convergence on each type of input information separately.Themean absolute percentage error(MAPE)of the proposedmethod is 0.463%,a reduction of 0.294% compared to the benchmark attention algorithm.Additionally,the market backtesting results indicate that the return was 24.560%,an improvement of 8.202% compared to the benchmark algorithm.These results suggest that themarket trading strategy based on thismethod has the potential to improve trading profits.

Posture Detection of Heart Disease Using Multi-Head Attention Vision Hybrid(MHAVH)Model

Cardiovascular disease is the leading cause of death globally.This disease causes loss of heart muscles and is also responsible for the death of heart cells,sometimes damaging their functionality.A person’s life may depend on receiving timely assistance as soon as possible.Thus,minimizing the death ratio can be achieved by early detection of heart attack(HA)symptoms.In the United States alone,an estimated 610,000 people die fromheart attacks each year,accounting for one in every four fatalities.However,by identifying and reporting heart attack symptoms early on,it is possible to reduce damage and save many lives significantly.Our objective is to devise an algorithm aimed at helping individuals,particularly elderly individuals living independently,to safeguard their lives.To address these challenges,we employ deep learning techniques.We have utilized a vision transformer(ViT)to address this problem.However,it has a significant overhead cost due to its memory consumption and computational complexity because of scaling dot-product attention.Also,since transformer performance typically relies on large-scale or adequate data,adapting ViT for smaller datasets is more challenging.In response,we propose a three-in-one steam model,theMulti-Head Attention Vision Hybrid(MHAVH).Thismodel integrates a real-time posture recognition framework to identify chest pain postures indicative of heart attacks using transfer learning techniques,such as ResNet-50 and VGG-16,renowned for their robust feature extraction capabilities.By incorporatingmultiple heads into the vision transformer to generate additional metrics and enhance heart-detection capabilities,we leverage a 2019 posture-based dataset comprising RGB images,a novel creation by the author that marks the first dataset tailored for posture-based heart attack detection.Given the limited online data availability,we segmented this dataset into gender categories(male and female)and conducted testing on both segmented and original datasets.The training accuracy of our model reached an impressive 99.77%.Upon testing,the accuracy for male and female datasets was recorded at 92.87%and 75.47%,respectively.The combined dataset accuracy is 93.96%,showcasing a commendable performance overall.Our proposed approach demonstrates versatility in accommodating small and large datasets,offering promising prospects for real-world applications.

Multi-Head Attention Spatial-Temporal Graph Neural Networks for Traffic Forecasting

Accurate traffic prediction is crucial for an intelligent traffic system (ITS). However, the excessive non-linearity and complexity of the spatial-temporal correlation in traffic flow severely limit the prediction accuracy of most existing models, which simply stack temporal and spatial modules and fail to capture spatial-temporal features effectively. To improve the prediction accuracy, a multi-head attention spatial-temporal graph neural network (MSTNet) is proposed in this paper. First, the traffic data is decomposed into unique time spans that conform to positive rules, and valuable traffic node attributes are mined through an adaptive graph structure. Second, time and spatial features are captured using a multi-head attention spatial-temporal module. Finally, a multi-step prediction module is used to achieve future traffic condition prediction. Numerical experiments were conducted on an open-source dataset, and the results demonstrate that MSTNet performs well in spatial-temporal feature extraction and achieves more positive forecasting results than the baseline methods.

Using Recurrent Neural Network Structure and Multi-Head Attention with Convolution for Fraudulent Phone Text Recognition

Fraud cases have been a risk in society and people’s property security has been greatly threatened.In recent studies,many promising algorithms have been developed for social media offensive text recognition as well as sentiment analysis.These algorithms are also suitable for fraudulent phone text recognition.Compared to these tasks,the semantics of fraudulent words are more complex and more difficult to distinguish.Recurrent Neural Networks(RNN),the variants ofRNN,ConvolutionalNeuralNetworks(CNN),and hybrid neural networks to extract text features are used by most text classification research.However,a single network or a simple network combination cannot obtain rich characteristic knowledge of fraudulent phone texts relatively.Therefore,a new model is proposed in this paper.In the fraudulent phone text,the knowledge that can be learned by the model includes the sequence structure of sentences,the correlation between words,the correlation of contextual semantics,the feature of keywords in sentences,etc.The new model combines a bidirectional Long-Short Term Memory Neural Network(BiLSTM)or a bidirectional Gate Recurrent United(BiGRU)and a Multi-Head attention mechanism module with convolution.A normalization layer is added after the output of the final hidden layer.BiLSTM or BiGRU is used to build the encoding and decoding layer.Multi-head attention mechanism module with convolution(MHAC)enhances the ability of the model to learn global interaction information and multi-granularity local interaction information in fraudulent sentences.A fraudulent phone text dataset is produced by us in this paper.The THUCNews data sets and fraudulent phone text data sets are used in experiments.Experiment results show that compared with the baseline model,the proposed model(LMHACL)has the best experiment results in terms of Accuracy,Precision,Recall,and F1 score on the two data sets.And the performance indexes on fraudulent phone text data sets are all above 0.94.

Discharge Summaries Based Sentiment Detection Using Multi-Head Attention and CNN-BiGRU

Automatic extraction of the patient’s health information from the unstructured data concerning the discharge summary remains challenging.Discharge summary related documents contain various aspects of the patient health condition to examine the quality of treatment and thereby help improve decision-making in the medical field.Using a sentiment dictionary and feature engineering,the researchers primarily mine semantic text features.However,choosing and designing features requires a lot of manpower.The proposed approach is an unsupervised deep learning model that learns a set of clusters embedded in the latent space.A composite model including Active Learning(AL),Convolutional Neural Network(CNN),BiGRU,and Multi-Attention,called ACBMA in this research,is designed to measure the quality of treatment based on discharge summaries text sentiment detection.CNN is utilized for extracting the set of local features of text vectors.Then BiGRU network was utilized to extract the text’s global features to solve the issues that a single CNN cannot obtain global semantic information and the traditional Recurrent Neural Network(RNN)gradient disappearance.Experiments prove that the ACBMA method can demonstrate the effectiveness of the suggested method,achieve comparable results to state-of-arts methods in sentiment detection,and outperform them with accurate benchmarks.Finally,several algorithm studies ultimately determined that the ACBMA method is more precise for discharge summaries sentiment analysis.

Multi-Headed Deep Learning Models to Detect Abnormality of Alzheimer’s Patients

Worldwide,many elders are suffering from Alzheimer’s disease(AD).The elders with AD exhibit various abnormalities in their activities,such as sleep disturbances,wandering aimlessly,forgetting activities,etc.,which are the strong signs and symptoms of AD progression.Recognizing these symptoms in advance could assist to a quicker diagnosis and treatment and to prevent the progression of Disease to the next stage.The proposed method aims to detect the behavioral abnormalities found in Daily activities of AD patients(ADP)using wearables.In the proposed work,a publicly available dataset collected using wearables is applied.Currently,no real-world data is available to illustrate the daily activities of ADP.Hence,the proposed method has synthesized the wearables data according to the abnormal activities of ADP.In the proposed work,multi-headed(MH)architectures such as MH Convolutional Neural Network-Long Short-Term Mem-ory Network(CNN-LSTM),MH one-dimensional Convolutional Neural Network(1D-CNN)and MH two dimensional Convolutional Neural Network(2D-CNN)as well as conventional methods,namely CNN-LSTM,1D-CNN,2D-CNN have been implemented to model activity pattern.A multi-label prediction technique is applied to detect abnormal activities.The results obtained show that the proposed MH architectures achieve improved performance than the conventional methods.Moreover,the MH models for activity recognition perform better than the abnormality detection.

Deep Learning Based Efficient Crowd Counting System

Estimation of crowd count is becoming crucial nowadays,as it can help in security surveillance,crowd monitoring,and management for different events.It is challenging to determine the approximate crowd size from an image of the crowd’s density.Therefore in this research study,we proposed a multi-headed convolutional neural network architecture-based model for crowd counting,where we divided our proposed model into two main components:(i)the convolutional neural network,which extracts the feature across the whole image that is given to it as an input,and(ii)the multi-headed layers,which make it easier to evaluate density maps to estimate the number of people in the input image and determine their number in the crowd.We employed the available public benchmark crowd-counting datasets UCF CC 50 and ShanghaiTech parts A and B for model training and testing to validate the model’s performance.To analyze the results,we used two metrics Mean Absolute Error(MAE)and Mean Square Error(MSE),and compared the results of the proposed systems with the state-of-art models of crowd counting.The results show the superiority of the proposed system.

Multi-scale persistent spatiotemporal transformer for long-term urban traffic flow prediction

Long-term urban traffic flow prediction is an important task in the field of intelligent transportation,as it can help optimize traffic management and improve travel efficiency.To improve prediction accuracy,a crucial issue is how to model spatiotemporal dependency in urban traffic data.In recent years,many studies have adopted spatiotemporal neural networks to extract key information from traffic data.However,most models ignore the semantic spatial similarity between long-distance areas when mining spatial dependency.They also ignore the impact of predicted time steps on the next unpredicted time step for making long-term predictions.Moreover,these models lack a comprehensive data embedding process to represent complex spatiotemporal dependency.This paper proposes a multi-scale persistent spatiotemporal transformer(MSPSTT)model to perform accurate long-term traffic flow prediction in cities.MSPSTT adopts an encoder-decoder structure and incorporates temporal,periodic,and spatial features to fully embed urban traffic data to address these issues.The model consists of a spatiotemporal encoder and a spatiotemporal decoder,which rely on temporal,geospatial,and semantic space multi-head attention modules to dynamically extract temporal,geospatial,and semantic characteristics.The spatiotemporal decoder combines the context information provided by the encoder,integrates the predicted time step information,and is iteratively updated to learn the correlation between different time steps in the broader time range to improve the model’s accuracy for long-term prediction.Experiments on four public transportation datasets demonstrate that MSPSTT outperforms the existing models by up to 9.5%on three common metrics.

Detecting APT-Exploited Processes through Semantic Fusion and Interaction Prediction

Considering the stealthiness and persistence of Advanced Persistent Threats(APTs),system audit logs are leveraged in recent studies to construct system entity interaction provenance graphs to unveil threats in a host.Rule-based provenance graph APT detection approaches require elaborate rules and cannot detect unknown attacks,and existing learning-based approaches are limited by the lack of available APT attack samples or generally only perform graph-level anomaly detection,which requires lots of manual efforts to locate attack entities.This paper proposes an APT-exploited process detection approach called ThreatSniffer,which constructs the benign provenance graph from attack-free audit logs,fits normal system entity interactions and then detects APT-exploited processes by predicting the rationality of entity interactions.Firstly,ThreatSniffer understands system entities in terms of their file paths,interaction sequences,and the number distribution of interaction types and uses the multi-head self-attention mechanism to fuse these semantics.Then,based on the insight that APT-exploited processes interact with system entities they should not invoke,ThreatSniffer performs negative sampling on the benign provenance graph to generate non-existent edges,thus characterizing irrational entity interactions without requiring APT attack samples.At last,it employs a heterogeneous graph neural network as the interaction prediction model to aggregate the contextual information of entity interactions,and locate processes exploited by attackers,thereby achieving fine-grained APT detection.Evaluation results demonstrate that anomaly-based detection enables ThreatSniffer to identify all attack activities.Compared to the node-level APT detection method APT-KGL,ThreatSniffer achieves a 6.1%precision improvement because of its comprehensive understanding of entity semantics.

A New Industrial Intrusion Detection Method Based on CNN-BiLSTM

Nowadays,with the rapid development of industrial Internet technology,on the one hand,advanced industrial control systems(ICS)have improved industrial production efficiency.However,there are more and more cyber-attacks targeting industrial control systems.To ensure the security of industrial networks,intrusion detection systems have been widely used in industrial control systems,and deep neural networks have always been an effective method for identifying cyber attacks.Current intrusion detection methods still suffer from low accuracy and a high false alarm rate.Therefore,it is important to build a more efficient intrusion detection model.This paper proposes a hybrid deep learning intrusion detection method based on convolutional neural networks and bidirectional long short-term memory neural networks(CNN-BiLSTM).To address the issue of imbalanced data within the dataset and improve the model’s detection capabilities,the Synthetic Minority Over-sampling Technique-Edited Nearest Neighbors(SMOTE-ENN)algorithm is applied in the preprocessing phase.This algorithm is employed to generate synthetic instances for the minority class,simultaneously mitigating the impact of noise in the majority class.This approach aims to create a more equitable distribution of classes,thereby enhancing the model’s ability to effectively identify patterns in both minority and majority classes.In the experimental phase,the detection performance of the method is verified using two data sets.Experimental results show that the accuracy rate on the CICIDS-2017 data set reaches 97.7%.On the natural gas pipeline dataset collected by Lan Turnipseed from Mississippi State University in the United States,the accuracy rate also reaches 85.5%.

基于Multi-head Attention和Bi-LSTM的实体关系分类

关系分类是自然语言处理领域的一项重要任务,能够为知识图谱的构建、问答系统和信息检索等提供技术支持.与传统关系分类方法相比较,基于神经网络和注意力机制的关系分类模型在各种关系分类任务中都获得了更出色的表现.以往的模型大多采用单层注意力机制,特征表达相对单一.因此本文在已有研究基础上,引入多头注意力机制(Multi-head attention),旨在让模型从不同表示空间上获取关于句子更多层面的信息,提高模型的特征表达能力.同时在现有的词向量和位置向量作为网络输入的基础上,进一步引入依存句法特征和相对核心谓词依赖特征,其中依存句法特征包括当前词的依存关系值和所依赖的父节点位置,从而使模型进一步获取更多的文本句法信息.在SemEval-2010 任务8 数据集上的实验结果证明,该方法相较之前的深度学习模型,性能有进一步提高.

Sparse Autoencoder-based Multi-head Deep Neural Networks for Machinery Fault Diagnostics with Detection of Novelties

Supervised fault diagnosis typically assumes that all the types of machinery failures are known.However,in practice unknown types of defect,i.e.,novelties,may occur,whose detection is a challenging task.In this paper,a novel fault diagnostic method is developed for both diagnostics and detection of novelties.To this end,a sparse autoencoder-based multi-head Deep Neural Network(DNN)is presented to jointly learn a shared encoding representation for both unsupervised reconstruction and supervised classification of the monitoring data.The detection of novelties is based on the reconstruction error.Moreover,the computational burden is reduced by directly training the multi-head DNN with rectified linear unit activation function,instead of performing the pre-training and fine-tuning phases required for classical DNNs.The addressed method is applied to a benchmark bearing case study and to experimental data acquired from a delta 3D printer.The results show that its performance is satisfactory both in detection of novelties and fault diagnosis,outperforming other state-of-the-art methods.This research proposes a novel fault diagnostics method which can not only diagnose the known type of defect,but also detect unknown types of defects.

An iterative ITI cancellation method for multi-head multi-track bit-patterned magnetic recording systems

Bit-Pattemed Magnetic Recording(BPMR)is one of the emerging data storage technologies,which promises an Areal Density(AD)of about 4 Tb/in2.However,a major problem practically encountered in a BPMR system is Inter-Track Interference(ITI)that can deteriorate the overall system performance,especially at high ADs.This paper proposes an iterative ITI cancellation method for an m-head m-track BPMR system,which uses m heads to read m adjacent tracks and decodes them simultaneously.To cancel the ITI,we subtract the weighted readback signals of adjacent tracks,acting as the ITI signals,from the readback signal of the target track,before passing the refined readback signal to a turbo decoder.Then,the decoded data will be employed to reconstruct the ITI signal for the next turbo iteration.Experimental results indicate that the proposed system performs better than the conventional system that uses one head to read one data track.Furthermore,we also find out that the proposed system is more robust to media noise and track misregistration than the conventional system.

融合底层信息的电气工程领域神经机器翻译

针对目前主流的神经机器翻译模型Transformer内部结构单元堆叠而造成的底层信息丢失和多层单元输出信息偏差不同的问题,对其结构进行了改进,提出了一种融合底层信息的神经机器翻译模型。采用多种网络结构对源语言进行底层信息的特征提取,并采用残差连接的方式实现底层信息的向上传递。实验结果显示:融合底层信息后的翻译模型在电气工程领域内的双语评估研究(BLEU)值最多提升了2.47个百分点。

基于半监督多头网络的腰椎CT图像分割

针对医学图像分割任务中医学数据标注困难以及CT图像强度不均匀问题,提出一种基于半监督的多头分割网络SSMH-Net。SSMH-Net网络采用教师—学生训练架构,基于相同的分割模型V-Net,通过指数移动平均算法完成教师与学生模型的交互训练;采用Multi-Head方法估计模型预测的不确定性信息,指导分割模型在更可靠的目标中学习。在CTspine分割数据集上,SSMH-Net网络平均分割Dice系数达到95.70%,表现出较为优异的分割性能。

基于站点实时关联度的短时公交客流预测方法

为探究公交站点之间的关联度并对公交客流进行更精准的实时预测,本文提出基于Attention的交通预测核心算法(Traffic Forecast Model Based Attention,TFMA),结合数据预处理和站点信息编码完成基于站点实时关联度的短时公交客流预测方法。该方法首先创新性地提出了站点实时关联度,可实现对目标站点客流量更精准的预测;其次,在公交站点的编码信息中融入线路站点信息、客流变化率、天气、日期等关联因素;接着,该方法依靠Attention机制计算站点实时关联度;核心算法中使用multi-headed机制、增加通道和残差连接进一步提升预测能力;最后,以苏州市公交数据进行验证。结果显示:在准确率上,对比多元线性回归的53.8%、GRU(Gated Recurrent Unit)的66.9%和LightGBM(Light Gradient Boosting Machine)的81.2%,本文提出的基于站点实时关联度的短时公交客流预测方法的准确率在90%以上,表明该方法具备优秀的短时公交客流预测能力。

融合多头自注意力机制的中文分类方法

中文文本分类任务中,深度学习神经网络方法具有自动提取特征、特征表达能力强的优势,但其模型可解释性不强。提出了一种Text-CNN+Multi-Head Attention模型,引入多头自注意力机制克服Text-CNN可解释性的不足。首先采用Text-CNN神经网络,高效提取文本局部特征信息;然后通过引入多头自注意力机制,最大限度发挥Text-CNN的并行运算能力,强调文本序列全局信息的捕捉;最后在时间和空间上完成对文本信息的特征提取。实验结果表明,提出的模型较其他模型在保证运算速度的同时,准确率提升了1%~2%。

An Innovative Approach Utilizing Binary-View Transformer for Speech Recognition Task

The deep learning advancements have greatly improved the performance of speech recognition systems,and most recent systems are based on the Recurrent Neural Network(RNN).Overall,the RNN works fine with the small sequence data,but suffers from the gradient vanishing problem in case of large sequence.The transformer networks have neutralized this issue and have shown state-of-the-art results on sequential or speech-related data.Generally,in speech recognition,the input audio is converted into an image using Mel-spectrogram to illustrate frequencies and intensities.The image is classified by the machine learning mechanism to generate a classification transcript.However,the audio frequency in the image has low resolution and causing inaccurate predictions.This paper presents a novel end-to-end binary view transformer-based architecture for speech recognition to cope with the frequency resolution problem.Firstly,the input audio signal is transformed into a 2D image using Mel-spectrogram.Secondly,the modified universal transformers utilize the multi-head attention to derive contextual information and derive different speech-related features.Moreover,a feedforward neural network is also deployed for classification.The proposed system has generated robust results on Google’s speech command dataset with an accuracy of 95.16%and with minimal loss.The binary-view transformer eradicates the eventuality of the over-fitting problem by deploying a multiview mechanism to diversify the input data,and multi-head attention captures multiple contexts from the data’s feature map.

Predictive Model of Live Shopping Interest Degree Based on Eye Movement Characteristics and Deep Factorization Machine

In the live broadcast process,eye movement characteristics can reflect people’s attention to the product.However,the existing interest degree predictive model research does not consider the eye movement characteristics.In order to obtain the users’interest in the product more effectively,we will consider the key eye movement indicators.We first collect eye movement characteristics based on the self-developed data processing algorithm fast discriminative model prediction for tracking(FDIMP),and then we add data dimensions to the original data set through information filling.In addition,we apply the deep factorization machine(DeepFM)architecture to simultaneously learn the combination of low-level and high-level features.In order to effectively learn important features and emphasize relatively important features,the multi-head attention mechanism is applied in the interest model.The experimental results on the public data set Criteo show that,compared with the original DeepFM algorithm,the area under curve(AUC)value was improved by up to 9.32%.