Application of the improved dung beetle optimizer,muti-head attention and hybrid deep learning algorithms to groundwater depth prediction in the Ningxia area,China

Due to the lack of accurate data and complex parameterization,the prediction of groundwater depth is a chal-lenge for numerical models.Machine learning can effectively solve this issue and has been proven useful in the prediction of groundwater depth in many areas.In this study,two new models are applied to the prediction of groundwater depth in the Ningxia area,China.The two models combine the improved dung beetle optimizer(DBO)algorithm with two deep learning models:The Multi-head Attention-Convolution Neural Network-Long Short Term Memory networks(MH-CNN-LSTM)and the Multi-head Attention-Convolution Neural Network-Gated Recurrent Unit(MH-CNN-GRU).The models with DBO show better prediction performance,with larger R(correlation coefficient),RPD(residual prediction deviation),and lower RMSE(root-mean-square error).Com-pared with the models with the original DBO,the R and RPD of models with the improved DBO increase by over 1.5%,and the RMSE decreases by over 1.8%,indicating better prediction results.In addition,compared with the multiple linear regression model,a traditional statistical model,deep learning models have better prediction performance.

An Intelligent Framework for Resilience Recovery of FANETs with Spatio-Temporal Aggregation and Multi-Head Attention Mechanism

Due to the time-varying topology and possible disturbances in a conflict environment,it is still challenging to maintain the mission performance of flying Ad hoc networks(FANET),which limits the application of Unmanned Aerial Vehicle(UAV)swarms in harsh environments.This paper proposes an intelligent framework to quickly recover the cooperative coveragemission by aggregating the historical spatio-temporal network with the attention mechanism.The mission resilience metric is introduced in conjunction with connectivity and coverage status information to simplify the optimization model.A spatio-temporal node pooling method is proposed to ensure all node location features can be updated after destruction by capturing the temporal network structure.Combined with the corresponding Laplacian matrix as the hyperparameter,a recovery algorithm based on the multi-head attention graph network is designed to achieve rapid recovery.Simulation results showed that the proposed framework can facilitate rapid recovery of the connectivity and coverage more effectively compared to the existing studies.The results demonstrate that the average connectivity and coverage results is improved by 17.92%and 16.96%,respectively compared with the state-of-the-art model.Furthermore,by the ablation study,the contributions of each different improvement are compared.The proposed model can be used to support resilient network design for real-time mission execution.

Structured Multi-Head Attention Stock Index Prediction Method Based Adaptive Public Opinion Sentiment Vector

The present study examines the impact of short-term public opinion sentiment on the secondary market,with a focus on the potential for such sentiment to cause dramatic stock price fluctuations and increase investment risk.The quantification of investment sentiment indicators and the persistent analysis of their impact has been a complex and significant area of research.In this paper,a structured multi-head attention stock index prediction method based adaptive public opinion sentiment vector is proposed.The proposedmethod utilizes an innovative approach to transform numerous investor comments on social platforms over time into public opinion sentiment vectors expressing complex sentiments.It then analyzes the continuous impact of these vectors on the market through the use of aggregating techniques and public opinion data via a structured multi-head attention mechanism.The experimental results demonstrate that the public opinion sentiment vector can provide more comprehensive feedback on market sentiment than traditional sentiment polarity analysis.Furthermore,the multi-head attention mechanism is shown to improve prediction accuracy through attention convergence on each type of input information separately.Themean absolute percentage error(MAPE)of the proposedmethod is 0.463%,a reduction of 0.294% compared to the benchmark attention algorithm.Additionally,the market backtesting results indicate that the return was 24.560%,an improvement of 8.202% compared to the benchmark algorithm.These results suggest that themarket trading strategy based on thismethod has the potential to improve trading profits.

Posture Detection of Heart Disease Using Multi-Head Attention Vision Hybrid(MHAVH)Model

Cardiovascular disease is the leading cause of death globally.This disease causes loss of heart muscles and is also responsible for the death of heart cells,sometimes damaging their functionality.A person’s life may depend on receiving timely assistance as soon as possible.Thus,minimizing the death ratio can be achieved by early detection of heart attack(HA)symptoms.In the United States alone,an estimated 610,000 people die fromheart attacks each year,accounting for one in every four fatalities.However,by identifying and reporting heart attack symptoms early on,it is possible to reduce damage and save many lives significantly.Our objective is to devise an algorithm aimed at helping individuals,particularly elderly individuals living independently,to safeguard their lives.To address these challenges,we employ deep learning techniques.We have utilized a vision transformer(ViT)to address this problem.However,it has a significant overhead cost due to its memory consumption and computational complexity because of scaling dot-product attention.Also,since transformer performance typically relies on large-scale or adequate data,adapting ViT for smaller datasets is more challenging.In response,we propose a three-in-one steam model,theMulti-Head Attention Vision Hybrid(MHAVH).Thismodel integrates a real-time posture recognition framework to identify chest pain postures indicative of heart attacks using transfer learning techniques,such as ResNet-50 and VGG-16,renowned for their robust feature extraction capabilities.By incorporatingmultiple heads into the vision transformer to generate additional metrics and enhance heart-detection capabilities,we leverage a 2019 posture-based dataset comprising RGB images,a novel creation by the author that marks the first dataset tailored for posture-based heart attack detection.Given the limited online data availability,we segmented this dataset into gender categories(male and female)and conducted testing on both segmented and original datasets.The training accuracy of our model reached an impressive 99.77%.Upon testing,the accuracy for male and female datasets was recorded at 92.87%and 75.47%,respectively.The combined dataset accuracy is 93.96%,showcasing a commendable performance overall.Our proposed approach demonstrates versatility in accommodating small and large datasets,offering promising prospects for real-world applications.

Multi-Head Attention Spatial-Temporal Graph Neural Networks for Traffic Forecasting

Accurate traffic prediction is crucial for an intelligent traffic system (ITS). However, the excessive non-linearity and complexity of the spatial-temporal correlation in traffic flow severely limit the prediction accuracy of most existing models, which simply stack temporal and spatial modules and fail to capture spatial-temporal features effectively. To improve the prediction accuracy, a multi-head attention spatial-temporal graph neural network (MSTNet) is proposed in this paper. First, the traffic data is decomposed into unique time spans that conform to positive rules, and valuable traffic node attributes are mined through an adaptive graph structure. Second, time and spatial features are captured using a multi-head attention spatial-temporal module. Finally, a multi-step prediction module is used to achieve future traffic condition prediction. Numerical experiments were conducted on an open-source dataset, and the results demonstrate that MSTNet performs well in spatial-temporal feature extraction and achieves more positive forecasting results than the baseline methods.

基于Multi-head Attention和Bi-LSTM的实体关系分类

关系分类是自然语言处理领域的一项重要任务,能够为知识图谱的构建、问答系统和信息检索等提供技术支持.与传统关系分类方法相比较,基于神经网络和注意力机制的关系分类模型在各种关系分类任务中都获得了更出色的表现.以往的模型大多采用单层注意力机制,特征表达相对单一.因此本文在已有研究基础上,引入多头注意力机制(Multi-head attention),旨在让模型从不同表示空间上获取关于句子更多层面的信息,提高模型的特征表达能力.同时在现有的词向量和位置向量作为网络输入的基础上,进一步引入依存句法特征和相对核心谓词依赖特征,其中依存句法特征包括当前词的依存关系值和所依赖的父节点位置,从而使模型进一步获取更多的文本句法信息.在SemEval-2010 任务8 数据集上的实验结果证明,该方法相较之前的深度学习模型,性能有进一步提高.

Multi-head attention-based long short-term memory model for speech emotion recognition

To fully make use of information from different representation subspaces,a multi-head attention-based long short-term memory(LSTM)model is proposed in this study for speech emotion recognition(SER).The proposed model uses frame-level features and takes the temporal information of emotion speech as the input of the LSTM layer.Here,a multi-head time-dimension attention(MHTA)layer was employed to linearly project the output of the LSTM layer into different subspaces for the reduced-dimension context vectors.To provide relative vital information from other dimensions,the output of MHTA,the output of feature-dimension attention,and the last time-step output of LSTM were utilized to form multiple context vectors as the input of the fully connected layer.To improve the performance of multiple vectors,feature-dimension attention was employed for the all-time output of the first LSTM layer.The proposed model was evaluated on the eNTERFACE and GEMEP corpora,respectively.The results indicate that the proposed model outperforms LSTM by 14.6%and 10.5%for eNTERFACE and GEMEP,respectively,proving the effectiveness of the proposed model in SER tasks.

Using Recurrent Neural Network Structure and Multi-Head Attention with Convolution for Fraudulent Phone Text Recognition

Fraud cases have been a risk in society and people’s property security has been greatly threatened.In recent studies,many promising algorithms have been developed for social media offensive text recognition as well as sentiment analysis.These algorithms are also suitable for fraudulent phone text recognition.Compared to these tasks,the semantics of fraudulent words are more complex and more difficult to distinguish.Recurrent Neural Networks(RNN),the variants ofRNN,ConvolutionalNeuralNetworks(CNN),and hybrid neural networks to extract text features are used by most text classification research.However,a single network or a simple network combination cannot obtain rich characteristic knowledge of fraudulent phone texts relatively.Therefore,a new model is proposed in this paper.In the fraudulent phone text,the knowledge that can be learned by the model includes the sequence structure of sentences,the correlation between words,the correlation of contextual semantics,the feature of keywords in sentences,etc.The new model combines a bidirectional Long-Short Term Memory Neural Network(BiLSTM)or a bidirectional Gate Recurrent United(BiGRU)and a Multi-Head attention mechanism module with convolution.A normalization layer is added after the output of the final hidden layer.BiLSTM or BiGRU is used to build the encoding and decoding layer.Multi-head attention mechanism module with convolution(MHAC)enhances the ability of the model to learn global interaction information and multi-granularity local interaction information in fraudulent sentences.A fraudulent phone text dataset is produced by us in this paper.The THUCNews data sets and fraudulent phone text data sets are used in experiments.Experiment results show that compared with the baseline model,the proposed model(LMHACL)has the best experiment results in terms of Accuracy,Precision,Recall,and F1 score on the two data sets.And the performance indexes on fraudulent phone text data sets are all above 0.94.

Discharge Summaries Based Sentiment Detection Using Multi-Head Attention and CNN-BiGRU

Automatic extraction of the patient’s health information from the unstructured data concerning the discharge summary remains challenging.Discharge summary related documents contain various aspects of the patient health condition to examine the quality of treatment and thereby help improve decision-making in the medical field.Using a sentiment dictionary and feature engineering,the researchers primarily mine semantic text features.However,choosing and designing features requires a lot of manpower.The proposed approach is an unsupervised deep learning model that learns a set of clusters embedded in the latent space.A composite model including Active Learning(AL),Convolutional Neural Network(CNN),BiGRU,and Multi-Attention,called ACBMA in this research,is designed to measure the quality of treatment based on discharge summaries text sentiment detection.CNN is utilized for extracting the set of local features of text vectors.Then BiGRU network was utilized to extract the text’s global features to solve the issues that a single CNN cannot obtain global semantic information and the traditional Recurrent Neural Network(RNN)gradient disappearance.Experiments prove that the ACBMA method can demonstrate the effectiveness of the suggested method,achieve comparable results to state-of-arts methods in sentiment detection,and outperform them with accurate benchmarks.Finally,several algorithm studies ultimately determined that the ACBMA method is more precise for discharge summaries sentiment analysis.

Study on the fusion emotion classification of multiple characteristics based on attention mechanism

The current research on emotional classification uses many methods that combine the attention mechanism with neural networks.However,the effect is unsatisfactory when dealing with complex text.An emotional classification model is proposed,which combines multi-head attention(MHA)with improved structured-self attention(SSA).The model makes several different linear transformations of input by introducing MHA mechanism and can extract more comprehensive high-level phrase representation features from the word embedded vector.Meanwhile,it can realize the parallelization calculation and ensure the training speed of the model.The improved SSA structure uses matrices to represent different parts of a sentence to extract local key information,to ensure that the degree of dependence between words is not affected by time and sentence length,and generate the overall semantics of the sentence.Experiment results show that the current model effectively obtains global structural information and improves classification accuracy.

A GAN-EfficientNet-Based Traceability Method for Malicious Code Variant Families

Due to the diversity and unpredictability of changes in malicious code,studying the traceability of variant families remains challenging.In this paper,we propose a GAN-EfficientNetV2-based method for tracing families of malicious code variants.This method leverages the similarity in layouts and textures between images of malicious code variants from the same source and their original family of malicious code images.The method includes a lightweight classifier and a simulator.The classifier utilizes the enhanced EfficientNetV2 to categorize malicious code images and can be easily deployed on mobile,embedded,and other devices.The simulator utilizes an enhanced generative adversarial network to simulate different variants of malicious code and generates datasets to validate the model’s performance.This process helps identify model vulnerabilities and security risks,facilitating model enhancement and development.The classifier achieves 98.61%and 97.59%accuracy on the MMCC dataset and Malevis dataset,respectively.The simulator’s generated image of malicious code variants has an FID value of 155.44 and an IS value of 1.72±0.42.The classifier’s accuracy for tracing the family of malicious code variants is as high as 90.29%,surpassing that of mainstream neural network models.This meets the current demand for high generalization and anti-obfuscation abilities in malicious code classification models due to the rapid evolution of malicious code.

基于双分支多头注意力的场景图生成方法

针对已有场景图生成模型获取上下文信息有限的问题,提出一种有效的上下文融合模块,即双分支多头注意力(dual-stream multi-head attention, DMA)模块,并将DMA分别用于物体分类阶段和关系分类阶段,基于此提出基于双分支多头注意力的场景图生成网络(dual-stream multi-head attention-based scene graph generation network, DMA-Net)。该网络由目标检测、物体语义解析和关系语义解析3个模块组成。首先,通过目标检测模块定位图像中的物体并提取物体特征;其次,使用物体语义解析模块中的节点双分支多头注意力(object dual-stream multi-head attention, O-DMA)获取融合了节点上下文的特征,该特征经过物体语义解码器获得物体类别标签;最后,通过关系语义解析模块中的边双分支多头注意力(relationship dual-stream multi-head attention, R-DMA)输出融合了边上下文的特征,该特征经过关系语义解码器输出关系类别标签。在公开的视觉基因组(visual genome, VG)数据集上分别计算了DMA-Net针对场景图检测、场景图分类和谓词分类3个子任务的图约束召回率和无图约束召回率,并与主流的场景图生成方法进行比较。实验结果表明,所提出的方法能够充分挖掘场景中的上下文信息,基于上下文增强的特征表示有效提升了场景图生成任务的精度。

Deep Learning Based Efficient Crowd Counting System

Estimation of crowd count is becoming crucial nowadays,as it can help in security surveillance,crowd monitoring,and management for different events.It is challenging to determine the approximate crowd size from an image of the crowd’s density.Therefore in this research study,we proposed a multi-headed convolutional neural network architecture-based model for crowd counting,where we divided our proposed model into two main components:(i)the convolutional neural network,which extracts the feature across the whole image that is given to it as an input,and(ii)the multi-headed layers,which make it easier to evaluate density maps to estimate the number of people in the input image and determine their number in the crowd.We employed the available public benchmark crowd-counting datasets UCF CC 50 and ShanghaiTech parts A and B for model training and testing to validate the model’s performance.To analyze the results,we used two metrics Mean Absolute Error(MAE)and Mean Square Error(MSE),and compared the results of the proposed systems with the state-of-art models of crowd counting.The results show the superiority of the proposed system.

Multi-scale persistent spatiotemporal transformer for long-term urban traffic flow prediction

Long-term urban traffic flow prediction is an important task in the field of intelligent transportation,as it can help optimize traffic management and improve travel efficiency.To improve prediction accuracy,a crucial issue is how to model spatiotemporal dependency in urban traffic data.In recent years,many studies have adopted spatiotemporal neural networks to extract key information from traffic data.However,most models ignore the semantic spatial similarity between long-distance areas when mining spatial dependency.They also ignore the impact of predicted time steps on the next unpredicted time step for making long-term predictions.Moreover,these models lack a comprehensive data embedding process to represent complex spatiotemporal dependency.This paper proposes a multi-scale persistent spatiotemporal transformer(MSPSTT)model to perform accurate long-term traffic flow prediction in cities.MSPSTT adopts an encoder-decoder structure and incorporates temporal,periodic,and spatial features to fully embed urban traffic data to address these issues.The model consists of a spatiotemporal encoder and a spatiotemporal decoder,which rely on temporal,geospatial,and semantic space multi-head attention modules to dynamically extract temporal,geospatial,and semantic characteristics.The spatiotemporal decoder combines the context information provided by the encoder,integrates the predicted time step information,and is iteratively updated to learn the correlation between different time steps in the broader time range to improve the model’s accuracy for long-term prediction.Experiments on four public transportation datasets demonstrate that MSPSTT outperforms the existing models by up to 9.5%on three common metrics.

NeurstrucEnergy:A bi-directional GNN model for energy prediction of neural networks in IoT

A significant demand rises for energy-efficient deep neural networks to support power-limited embedding devices with successful deep learning applications in IoT and edge computing fields.An accurate energy prediction approach is critical to provide measurement and lead optimization direction.However,the current energy prediction approaches lack accuracy and generalization ability due to the lack of research on the neural network structure and the excessive reliance on customized training dataset.This paper presents a novel energy prediction model,NeurstrucEnergy.NeurstrucEnergy treats neural networks as directed graphs and applies a bi-directional graph neural network training on a randomly generated dataset to extract structural features for energy prediction.NeurstrucEnergy has advantages over linear approaches because the bi-directional graph neural network collects structural features from each layer's parents and children.Experimental results show that NeurstrucEnergy establishes state-of-the-art results with mean absolute percentage error of 2.60%.We also evaluate NeurstrucEnergy in a randomly generated dataset,achieving the mean absolute percentage error of 4.83%over 10 typical convolutional neural networks in recent years and 7 efficient convolutional neural networks created by neural architecture search.Our code is available at https://github.com/NEUSoftGreenAI/NeurstrucEnergy.git.

Short-term load forecasting model based on gated recurrent unit and multi-head attention

Short-term load forecasting(STLF)plays a crucial role in the smart grid.However,it is challenging to capture the long-time dependence and the nonlinear relationship due to the comprehensive fluctuations of the electrical load.In this paper,an STLF model based on gated recurrent unit and multi-head attention(GRU-MA)is proposed to address the aforementioned problems.The proposed model accommodates the time series and nonlinear relationship of load data through gated recurrent unit(GRU)and exploits multi-head attention(MA)to learn the decisive features and long-term dependencies.Additionally,the proposed model is compared with the support vector regression(SVR)model,the recurrent neural network and multi-head attention(RNN-MA)model,the long short-term memory and multi-head attention(LSTM-MA)model,the GRU model,and the temporal convolutional network(TCN)model using the public dataset of the Global Energy Forecasting Competition 2014(GEFCOM2014).The results demonstrate that the GRU-MA model has the best prediction accuracy.

基于多尺度特征融合的无人机姿态估计方法

无人机已广泛应用于物资运输、应急救援、泵站巡检、水利工程项目巡检等军事和民用领域,其中无人机定位技术是其实现自主飞行及实际应用的关键。在泵站及地下管网等复杂环境中,卫星信号微弱或中断,给无人机姿态的获取带来严峻挑战。针对此问题,文章结合实际工程案例,提出了一种基于多尺度特征融合的无人机姿态估计方法,该方法能够直接从输入的RGB(红、绿、蓝)图像中回归出无人机的6自由度姿态信息。该方法利用CNN(卷积神经网络)提取图像中的多尺度空间特征;通过两个并行的AttnLSTM(注意力机制长短期记忆网络)从两个不同方向处理数据,实现结构化降维并提升模型的泛化能力;同时,引入Multi-Head Attention(多头注意力)机制,进一步提炼关键信息,实现多尺度特征的有效融合。实验结果表明,该模型在公开数据集上表现出色,能够准确估计无人机的姿态信息。

Fiber communication receiver models based on the multi-head attention mechanism

In this paper,an artificial-intelligence-based fiber communication receiver model is put forward.With the multi-head attention mechanism it contains,this model can extract crucial patterns and map the transmitted signals into the bit stream.Once appropriately trained,it can obtain the ability to restore the information from the signals whose transmission distances range from 0 to 100 km,signal-to-noise ratios range from 0 to 20 dB,modulation formats range from OOK to PAM4,and symbol rates range from 10 to 40 GBaud.The validity of the model is numerically demonstrated via MATLAB and Pytorch scenarios and compared with traditional communication receivers.

基于站点实时关联度的短时公交客流预测方法

为探究公交站点之间的关联度并对公交客流进行更精准的实时预测,本文提出基于Attention的交通预测核心算法(Traffic Forecast Model Based Attention,TFMA),结合数据预处理和站点信息编码完成基于站点实时关联度的短时公交客流预测方法。该方法首先创新性地提出了站点实时关联度,可实现对目标站点客流量更精准的预测;其次,在公交站点的编码信息中融入线路站点信息、客流变化率、天气、日期等关联因素;接着,该方法依靠Attention机制计算站点实时关联度;核心算法中使用multi-headed机制、增加通道和残差连接进一步提升预测能力;最后,以苏州市公交数据进行验证。结果显示:在准确率上,对比多元线性回归的53.8%、GRU(Gated Recurrent Unit)的66.9%和LightGBM(Light Gradient Boosting Machine)的81.2%,本文提出的基于站点实时关联度的短时公交客流预测方法的准确率在90%以上,表明该方法具备优秀的短时公交客流预测能力。

人自称、人被称与物被称

本文首先观察汉语"我"(的)丰富的变体这一自关心现象,并设置"人自称、人被称、物被称"三元并存范畴,其意义在于:(1)彰显人对自身生存状态的自关心,人自称的高度复杂性使我们认识到人的高度复杂性。(2)彰显人对自身生存状态的自关心、对他人生存状态的他关心、对物的他关心之间的差异。这一差异不仅提示人自称对语言的复杂诉求终归可以看成人对自己的优先彰显,而且顺便解释Frege关于"呈现方式与认知内容"不能解释的人自称的(超)多名(多变体)现象。(3)研究"我"变体可与海德格尔的"此在"相呼应。"‘我’是此在的本质规定性。"在Being and Time里,人的优先地位以海德格尔式的阐释方式实现,在汉语里则以人自称的方式实现。人自称正是在进行"对自己存在的解说"!假如某物在特定语言中的指称方式多有变体,该物的"存在和出场"比他物得到更多张扬。人对世界一人一事的称呼与描述,其实是以自己的眼光干涉其中的。