Posture Detection of Heart Disease Using Multi-Head Attention Vision Hybrid(MHAVH)Model

Cardiovascular disease is the leading cause of death globally.This disease causes loss of heart muscles and is also responsible for the death of heart cells,sometimes damaging their functionality.A person’s life may depend on receiving timely assistance as soon as possible.Thus,minimizing the death ratio can be achieved by early detection of heart attack(HA)symptoms.In the United States alone,an estimated 610,000 people die fromheart attacks each year,accounting for one in every four fatalities.However,by identifying and reporting heart attack symptoms early on,it is possible to reduce damage and save many lives significantly.Our objective is to devise an algorithm aimed at helping individuals,particularly elderly individuals living independently,to safeguard their lives.To address these challenges,we employ deep learning techniques.We have utilized a vision transformer(ViT)to address this problem.However,it has a significant overhead cost due to its memory consumption and computational complexity because of scaling dot-product attention.Also,since transformer performance typically relies on large-scale or adequate data,adapting ViT for smaller datasets is more challenging.In response,we propose a three-in-one steam model,theMulti-Head Attention Vision Hybrid(MHAVH).Thismodel integrates a real-time posture recognition framework to identify chest pain postures indicative of heart attacks using transfer learning techniques,such as ResNet-50 and VGG-16,renowned for their robust feature extraction capabilities.By incorporatingmultiple heads into the vision transformer to generate additional metrics and enhance heart-detection capabilities,we leverage a 2019 posture-based dataset comprising RGB images,a novel creation by the author that marks the first dataset tailored for posture-based heart attack detection.Given the limited online data availability,we segmented this dataset into gender categories(male and female)and conducted testing on both segmented and original datasets.The training accuracy of our model reached an impressive 99.77%.Upon testing,the accuracy for male and female datasets was recorded at 92.87%and 75.47%,respectively.The combined dataset accuracy is 93.96%,showcasing a commendable performance overall.Our proposed approach demonstrates versatility in accommodating small and large datasets,offering promising prospects for real-world applications.

An Intelligent Framework for Resilience Recovery of FANETs with Spatio-Temporal Aggregation and Multi-Head Attention Mechanism

Due to the time-varying topology and possible disturbances in a conflict environment,it is still challenging to maintain the mission performance of flying Ad hoc networks(FANET),which limits the application of Unmanned Aerial Vehicle(UAV)swarms in harsh environments.This paper proposes an intelligent framework to quickly recover the cooperative coveragemission by aggregating the historical spatio-temporal network with the attention mechanism.The mission resilience metric is introduced in conjunction with connectivity and coverage status information to simplify the optimization model.A spatio-temporal node pooling method is proposed to ensure all node location features can be updated after destruction by capturing the temporal network structure.Combined with the corresponding Laplacian matrix as the hyperparameter,a recovery algorithm based on the multi-head attention graph network is designed to achieve rapid recovery.Simulation results showed that the proposed framework can facilitate rapid recovery of the connectivity and coverage more effectively compared to the existing studies.The results demonstrate that the average connectivity and coverage results is improved by 17.92%and 16.96%,respectively compared with the state-of-the-art model.Furthermore,by the ablation study,the contributions of each different improvement are compared.The proposed model can be used to support resilient network design for real-time mission execution.

Structured Multi-Head Attention Stock Index Prediction Method Based Adaptive Public Opinion Sentiment Vector

The present study examines the impact of short-term public opinion sentiment on the secondary market,with a focus on the potential for such sentiment to cause dramatic stock price fluctuations and increase investment risk.The quantification of investment sentiment indicators and the persistent analysis of their impact has been a complex and significant area of research.In this paper,a structured multi-head attention stock index prediction method based adaptive public opinion sentiment vector is proposed.The proposedmethod utilizes an innovative approach to transform numerous investor comments on social platforms over time into public opinion sentiment vectors expressing complex sentiments.It then analyzes the continuous impact of these vectors on the market through the use of aggregating techniques and public opinion data via a structured multi-head attention mechanism.The experimental results demonstrate that the public opinion sentiment vector can provide more comprehensive feedback on market sentiment than traditional sentiment polarity analysis.Furthermore,the multi-head attention mechanism is shown to improve prediction accuracy through attention convergence on each type of input information separately.Themean absolute percentage error(MAPE)of the proposedmethod is 0.463%,a reduction of 0.294% compared to the benchmark attention algorithm.Additionally,the market backtesting results indicate that the return was 24.560%,an improvement of 8.202% compared to the benchmark algorithm.These results suggest that themarket trading strategy based on thismethod has the potential to improve trading profits.

Multi-Head Attention Spatial-Temporal Graph Neural Networks for Traffic Forecasting

Accurate traffic prediction is crucial for an intelligent traffic system (ITS). However, the excessive non-linearity and complexity of the spatial-temporal correlation in traffic flow severely limit the prediction accuracy of most existing models, which simply stack temporal and spatial modules and fail to capture spatial-temporal features effectively. To improve the prediction accuracy, a multi-head attention spatial-temporal graph neural network (MSTNet) is proposed in this paper. First, the traffic data is decomposed into unique time spans that conform to positive rules, and valuable traffic node attributes are mined through an adaptive graph structure. Second, time and spatial features are captured using a multi-head attention spatial-temporal module. Finally, a multi-step prediction module is used to achieve future traffic condition prediction. Numerical experiments were conducted on an open-source dataset, and the results demonstrate that MSTNet performs well in spatial-temporal feature extraction and achieves more positive forecasting results than the baseline methods.

Using Recurrent Neural Network Structure and Multi-Head Attention with Convolution for Fraudulent Phone Text Recognition

Fraud cases have been a risk in society and people’s property security has been greatly threatened.In recent studies,many promising algorithms have been developed for social media offensive text recognition as well as sentiment analysis.These algorithms are also suitable for fraudulent phone text recognition.Compared to these tasks,the semantics of fraudulent words are more complex and more difficult to distinguish.Recurrent Neural Networks(RNN),the variants ofRNN,ConvolutionalNeuralNetworks(CNN),and hybrid neural networks to extract text features are used by most text classification research.However,a single network or a simple network combination cannot obtain rich characteristic knowledge of fraudulent phone texts relatively.Therefore,a new model is proposed in this paper.In the fraudulent phone text,the knowledge that can be learned by the model includes the sequence structure of sentences,the correlation between words,the correlation of contextual semantics,the feature of keywords in sentences,etc.The new model combines a bidirectional Long-Short Term Memory Neural Network(BiLSTM)or a bidirectional Gate Recurrent United(BiGRU)and a Multi-Head attention mechanism module with convolution.A normalization layer is added after the output of the final hidden layer.BiLSTM or BiGRU is used to build the encoding and decoding layer.Multi-head attention mechanism module with convolution(MHAC)enhances the ability of the model to learn global interaction information and multi-granularity local interaction information in fraudulent sentences.A fraudulent phone text dataset is produced by us in this paper.The THUCNews data sets and fraudulent phone text data sets are used in experiments.Experiment results show that compared with the baseline model,the proposed model(LMHACL)has the best experiment results in terms of Accuracy,Precision,Recall,and F1 score on the two data sets.And the performance indexes on fraudulent phone text data sets are all above 0.94.

Discharge Summaries Based Sentiment Detection Using Multi-Head Attention and CNN-BiGRU

Automatic extraction of the patient’s health information from the unstructured data concerning the discharge summary remains challenging.Discharge summary related documents contain various aspects of the patient health condition to examine the quality of treatment and thereby help improve decision-making in the medical field.Using a sentiment dictionary and feature engineering,the researchers primarily mine semantic text features.However,choosing and designing features requires a lot of manpower.The proposed approach is an unsupervised deep learning model that learns a set of clusters embedded in the latent space.A composite model including Active Learning(AL),Convolutional Neural Network(CNN),BiGRU,and Multi-Attention,called ACBMA in this research,is designed to measure the quality of treatment based on discharge summaries text sentiment detection.CNN is utilized for extracting the set of local features of text vectors.Then BiGRU network was utilized to extract the text’s global features to solve the issues that a single CNN cannot obtain global semantic information and the traditional Recurrent Neural Network(RNN)gradient disappearance.Experiments prove that the ACBMA method can demonstrate the effectiveness of the suggested method,achieve comparable results to state-of-arts methods in sentiment detection,and outperform them with accurate benchmarks.Finally,several algorithm studies ultimately determined that the ACBMA method is more precise for discharge summaries sentiment analysis.

基于Multi-head Attention和Bi-LSTM的实体关系分类

关系分类是自然语言处理领域的一项重要任务,能够为知识图谱的构建、问答系统和信息检索等提供技术支持.与传统关系分类方法相比较,基于神经网络和注意力机制的关系分类模型在各种关系分类任务中都获得了更出色的表现.以往的模型大多采用单层注意力机制,特征表达相对单一.因此本文在已有研究基础上,引入多头注意力机制(Multi-head attention),旨在让模型从不同表示空间上获取关于句子更多层面的信息,提高模型的特征表达能力.同时在现有的词向量和位置向量作为网络输入的基础上,进一步引入依存句法特征和相对核心谓词依赖特征,其中依存句法特征包括当前词的依存关系值和所依赖的父节点位置,从而使模型进一步获取更多的文本句法信息.在SemEval-2010 任务8 数据集上的实验结果证明,该方法相较之前的深度学习模型,性能有进一步提高.

Multi-head attention-based long short-term memory model for speech emotion recognition

To fully make use of information from different representation subspaces,a multi-head attention-based long short-term memory(LSTM)model is proposed in this study for speech emotion recognition(SER).The proposed model uses frame-level features and takes the temporal information of emotion speech as the input of the LSTM layer.Here,a multi-head time-dimension attention(MHTA)layer was employed to linearly project the output of the LSTM layer into different subspaces for the reduced-dimension context vectors.To provide relative vital information from other dimensions,the output of MHTA,the output of feature-dimension attention,and the last time-step output of LSTM were utilized to form multiple context vectors as the input of the fully connected layer.To improve the performance of multiple vectors,feature-dimension attention was employed for the all-time output of the first LSTM layer.The proposed model was evaluated on the eNTERFACE and GEMEP corpora,respectively.The results indicate that the proposed model outperforms LSTM by 14.6%and 10.5%for eNTERFACE and GEMEP,respectively,proving the effectiveness of the proposed model in SER tasks.

Study on the fusion emotion classification of multiple characteristics based on attention mechanism

The current research on emotional classification uses many methods that combine the attention mechanism with neural networks.However,the effect is unsatisfactory when dealing with complex text.An emotional classification model is proposed,which combines multi-head attention(MHA)with improved structured-self attention(SSA).The model makes several different linear transformations of input by introducing MHA mechanism and can extract more comprehensive high-level phrase representation features from the word embedded vector.Meanwhile,it can realize the parallelization calculation and ensure the training speed of the model.The improved SSA structure uses matrices to represent different parts of a sentence to extract local key information,to ensure that the degree of dependence between words is not affected by time and sentence length,and generate the overall semantics of the sentence.Experiment results show that the current model effectively obtains global structural information and improves classification accuracy.

基于双分支多头注意力的场景图生成方法

针对已有场景图生成模型获取上下文信息有限的问题,提出一种有效的上下文融合模块,即双分支多头注意力(dual-stream multi-head attention, DMA)模块,并将DMA分别用于物体分类阶段和关系分类阶段,基于此提出基于双分支多头注意力的场景图生成网络(dual-stream multi-head attention-based scene graph generation network, DMA-Net)。该网络由目标检测、物体语义解析和关系语义解析3个模块组成。首先,通过目标检测模块定位图像中的物体并提取物体特征;其次,使用物体语义解析模块中的节点双分支多头注意力(object dual-stream multi-head attention, O-DMA)获取融合了节点上下文的特征,该特征经过物体语义解码器获得物体类别标签;最后,通过关系语义解析模块中的边双分支多头注意力(relationship dual-stream multi-head attention, R-DMA)输出融合了边上下文的特征,该特征经过关系语义解码器输出关系类别标签。在公开的视觉基因组(visual genome, VG)数据集上分别计算了DMA-Net针对场景图检测、场景图分类和谓词分类3个子任务的图约束召回率和无图约束召回率,并与主流的场景图生成方法进行比较。实验结果表明,所提出的方法能够充分挖掘场景中的上下文信息,基于上下文增强的特征表示有效提升了场景图生成任务的精度。

Deep Learning Based Efficient Crowd Counting System

Estimation of crowd count is becoming crucial nowadays,as it can help in security surveillance,crowd monitoring,and management for different events.It is challenging to determine the approximate crowd size from an image of the crowd’s density.Therefore in this research study,we proposed a multi-headed convolutional neural network architecture-based model for crowd counting,where we divided our proposed model into two main components:(i)the convolutional neural network,which extracts the feature across the whole image that is given to it as an input,and(ii)the multi-headed layers,which make it easier to evaluate density maps to estimate the number of people in the input image and determine their number in the crowd.We employed the available public benchmark crowd-counting datasets UCF CC 50 and ShanghaiTech parts A and B for model training and testing to validate the model’s performance.To analyze the results,we used two metrics Mean Absolute Error(MAE)and Mean Square Error(MSE),and compared the results of the proposed systems with the state-of-art models of crowd counting.The results show the superiority of the proposed system.

Aspect-Guided Multi-Graph Convolutional Networks for Aspect-based Sentiment Analysis

The Aspect-Based Sentiment Analysis(ABSA)task is designed to judge the sentiment polarity of a particular aspect in a review.Recent studies have proved that GCN can capture syntactic and semantic features from dependency graphs generated by dependency trees and semantic graphs generated by Multi-headed self-attention(MHSA).However,these approaches do not highlight the sentiment information associated with aspect in the syntactic and semantic graphs.We propose the Aspect-Guided Multi-Graph Convolutional Networks(AGGCN)for Aspect-Based Sentiment Classification.Specifically,we reconstruct two kinds of graphs,changing the weight of the dependency graph by distance from aspect and improving the semantic graph by Aspect-guided MHSA.For interactive learning of syntax and semantics,we dynamically fuse syntactic and semantic diagrams to generate syntactic-semantic graphs to learn emotional features jointly.In addition,Multi-dropout is added to solve the overftting of AGGCN in training.The experimental results on extensive datasets show that our model AGGCN achieves particularly advanced results and validates the effectiveness of the model.

Multi-scale persistent spatiotemporal transformer for long-term urban traffic flow prediction

Long-term urban traffic flow prediction is an important task in the field of intelligent transportation,as it can help optimize traffic management and improve travel efficiency.To improve prediction accuracy,a crucial issue is how to model spatiotemporal dependency in urban traffic data.In recent years,many studies have adopted spatiotemporal neural networks to extract key information from traffic data.However,most models ignore the semantic spatial similarity between long-distance areas when mining spatial dependency.They also ignore the impact of predicted time steps on the next unpredicted time step for making long-term predictions.Moreover,these models lack a comprehensive data embedding process to represent complex spatiotemporal dependency.This paper proposes a multi-scale persistent spatiotemporal transformer(MSPSTT)model to perform accurate long-term traffic flow prediction in cities.MSPSTT adopts an encoder-decoder structure and incorporates temporal,periodic,and spatial features to fully embed urban traffic data to address these issues.The model consists of a spatiotemporal encoder and a spatiotemporal decoder,which rely on temporal,geospatial,and semantic space multi-head attention modules to dynamically extract temporal,geospatial,and semantic characteristics.The spatiotemporal decoder combines the context information provided by the encoder,integrates the predicted time step information,and is iteratively updated to learn the correlation between different time steps in the broader time range to improve the model’s accuracy for long-term prediction.Experiments on four public transportation datasets demonstrate that MSPSTT outperforms the existing models by up to 9.5%on three common metrics.

NeurstrucEnergy:A bi-directional GNN model for energy prediction of neural networks in IoT

A significant demand rises for energy-efficient deep neural networks to support power-limited embedding devices with successful deep learning applications in IoT and edge computing fields.An accurate energy prediction approach is critical to provide measurement and lead optimization direction.However,the current energy prediction approaches lack accuracy and generalization ability due to the lack of research on the neural network structure and the excessive reliance on customized training dataset.This paper presents a novel energy prediction model,NeurstrucEnergy.NeurstrucEnergy treats neural networks as directed graphs and applies a bi-directional graph neural network training on a randomly generated dataset to extract structural features for energy prediction.NeurstrucEnergy has advantages over linear approaches because the bi-directional graph neural network collects structural features from each layer's parents and children.Experimental results show that NeurstrucEnergy establishes state-of-the-art results with mean absolute percentage error of 2.60%.We also evaluate NeurstrucEnergy in a randomly generated dataset,achieving the mean absolute percentage error of 4.83%over 10 typical convolutional neural networks in recent years and 7 efficient convolutional neural networks created by neural architecture search.Our code is available at https://github.com/NEUSoftGreenAI/NeurstrucEnergy.git.

Short-term load forecasting model based on gated recurrent unit and multi-head attention

Short-term load forecasting(STLF)plays a crucial role in the smart grid.However,it is challenging to capture the long-time dependence and the nonlinear relationship due to the comprehensive fluctuations of the electrical load.In this paper,an STLF model based on gated recurrent unit and multi-head attention(GRU-MA)is proposed to address the aforementioned problems.The proposed model accommodates the time series and nonlinear relationship of load data through gated recurrent unit(GRU)and exploits multi-head attention(MA)to learn the decisive features and long-term dependencies.Additionally,the proposed model is compared with the support vector regression(SVR)model,the recurrent neural network and multi-head attention(RNN-MA)model,the long short-term memory and multi-head attention(LSTM-MA)model,the GRU model,and the temporal convolutional network(TCN)model using the public dataset of the Global Energy Forecasting Competition 2014(GEFCOM2014).The results demonstrate that the GRU-MA model has the best prediction accuracy.

Fiber communication receiver models based on the multi-head attention mechanism

In this paper,an artificial-intelligence-based fiber communication receiver model is put forward.With the multi-head attention mechanism it contains,this model can extract crucial patterns and map the transmitted signals into the bit stream.Once appropriately trained,it can obtain the ability to restore the information from the signals whose transmission distances range from 0 to 100 km,signal-to-noise ratios range from 0 to 20 dB,modulation formats range from OOK to PAM4,and symbol rates range from 10 to 40 GBaud.The validity of the model is numerically demonstrated via MATLAB and Pytorch scenarios and compared with traditional communication receivers.

基于数据预处理和集成机器学习的锂离子电池剩余使用寿命预测

针对锂离子电池容量退化存在局部再生现象,导致单一模型预测不准确问题,提出了一种基于数据预处理和集成机器学习的锂离子电池剩余使用寿命(RUL)预测方法。首先,利用自适应噪声完全集成经验模态分解(CEEMDAN)算法将锂离子电池容量退化序列分解成波动数据和主趋势两个部分。然后,使用时域卷积网络(TCN)和多头注意力机制(MHA)的组合模型预测波动数据部分。对于主趋势部分,选择差分自回归移动平均模型(ARIMA)进行预测。最后,集成各预测结果得到锂离子电池剩余使用寿命的预测结果。以NASA公开的数据集进行验证,实验结果表明:四组电池的均方根误差不超过1.85%,平均绝对误差在1.25%以内。证明了所提出的多模型融合方法具有良好的预测性能和鲁棒性。

具有方面项和上下文表示的方面情感分析

针对预测特定方面情感极性时存在只依赖单一方面项而忽略了同一句子中方面项之间间情感依赖关系的问题,提出一种具有方面项和上下文表示的多层多跳记忆网络(AICR-M3net)。首先,通过双向门控循环单元(Bi-GRU)融合位置加权信息,并将隐藏层输出作为混合上下文编码层的输入以获取与上下文语义关联度更高的上下文表示;其次,引入多层多跳记忆网络(M3net)多次逐词匹配方面词和上下文,从而生成特定上下文的方面词向量;同时,建模特定方面项与句子中其他方面项的情感依赖性,从而引导特定方面项的上下文向量的生成。在Restaurant、Laptop和Twitter数据集上的实验结果表明,与AOA-MultiACIA(Attention-Over-Attention Multi-layer Aspect-Context Interactive Attention)相比,所提模型的分类准确率分别提高了1.34、3.05和2.02个百分点,F1值分别提高了3.90、3.78和2.94个百分点。以上验证了所提模型能更有效地处理上下文中多方面的混合信息,且在处理特定方面情感分类任务中具有一定的优势。

基于MHA⁃BiGRU的内河航道水位预测分析

针对山区内河航道水位预测技术难题,分析了影响航道水位预测的环境因素和技术难点,总结了当前航道水位预测模型的技术方法,提出一种新型的MHA⁃BiGRU航道水位预测模型,将多头注意力机制引入GRU模型,实现了模型对航道水位序列数据时间和空间等重要因素的特征权重划分,使模型聚焦影响航道水位变化的关键因素。以乌江下游航道为研究对象,通过建设水位和流速实时动态监测站,建立监测数据集,选取平均绝对误差(MAE)、纳什效率系数(NSE)和均方根误差(RMSE)等参数作为评价指标对该模型进行验证。结果表明,模型通过多头注意力机制和双向循环神经网络的应用,提升了航道水位预测性能;与传统的LSTM和GRU等经典时序预测模型相比,该模型具有更好的鲁棒性和更高的精度。将模型嵌入系统平台中进行示范应用,实现了航道水位的实时动态监测和中短期预测,具有较高的工程应用价值。

基于站点实时关联度的短时公交客流预测方法

为探究公交站点之间的关联度并对公交客流进行更精准的实时预测,本文提出基于Attention的交通预测核心算法(Traffic Forecast Model Based Attention,TFMA),结合数据预处理和站点信息编码完成基于站点实时关联度的短时公交客流预测方法。该方法首先创新性地提出了站点实时关联度,可实现对目标站点客流量更精准的预测;其次,在公交站点的编码信息中融入线路站点信息、客流变化率、天气、日期等关联因素;接着,该方法依靠Attention机制计算站点实时关联度;核心算法中使用multi-headed机制、增加通道和残差连接进一步提升预测能力;最后,以苏州市公交数据进行验证。结果显示:在准确率上,对比多元线性回归的53.8%、GRU(Gated Recurrent Unit)的66.9%和LightGBM(Light Gradient Boosting Machine)的81.2%,本文提出的基于站点实时关联度的短时公交客流预测方法的准确率在90%以上,表明该方法具备优秀的短时公交客流预测能力。