Multi-Head Attention Spatial-Temporal Graph Neural Networks for Traffic Forecasting

Accurate traffic prediction is crucial for an intelligent traffic system (ITS). However, the excessive non-linearity and complexity of the spatial-temporal correlation in traffic flow severely limit the prediction accuracy of most existing models, which simply stack temporal and spatial modules and fail to capture spatial-temporal features effectively. To improve the prediction accuracy, a multi-head attention spatial-temporal graph neural network (MSTNet) is proposed in this paper. First, the traffic data is decomposed into unique time spans that conform to positive rules, and valuable traffic node attributes are mined through an adaptive graph structure. Second, time and spatial features are captured using a multi-head attention spatial-temporal module. Finally, a multi-step prediction module is used to achieve future traffic condition prediction. Numerical experiments were conducted on an open-source dataset, and the results demonstrate that MSTNet performs well in spatial-temporal feature extraction and achieves more positive forecasting results than the baseline methods.

MSADCN:Multi-Scale Attentional Densely Connected Network for Automated Bone Age Assessment

Bone age assessment(BAA)helps doctors determine how a child’s bones grow and develop in clinical medicine.Traditional BAA methods rely on clinician expertise,leading to time-consuming predictions and inaccurate results.Most deep learning-based BAA methods feed the extracted critical points of images into the network by providing additional annotations.This operation is costly and subjective.To address these problems,we propose a multi-scale attentional densely connected network(MSADCN)in this paper.MSADCN constructs a multi-scale dense connectivity mechanism,which can avoid overfitting,obtain the local features effectively and prevent gradient vanishing even in limited training data.First,MSADCN designs multi-scale structures in the densely connected network to extract fine-grained features at different scales.Then,coordinate attention is embedded to focus on critical features and automatically locate the regions of interest(ROI)without additional annotation.In addition,to improve the model’s generalization,transfer learning is applied to train the proposed MSADCN on the public dataset IMDB-WIKI,and the obtained pre-trained weights are loaded onto the Radiological Society of North America(RSNA)dataset.Finally,label distribution learning(LDL)and expectation regression techniques are introduced into our model to exploit the correlation between hand bone images of different ages,which can obtain stable age estimates.Extensive experiments confirm that our model can converge more efficiently and obtain a mean absolute error(MAE)of 4.64 months,outperforming some state-of-the-art BAA methods.

The Short-Term Prediction ofWind Power Based on the Convolutional Graph Attention Deep Neural Network

The fluctuation of wind power affects the operating safety and power consumption of the electric power grid and restricts the grid connection of wind power on a large scale.Therefore,wind power forecasting plays a key role in improving the safety and economic benefits of the power grid.This paper proposes a wind power predicting method based on a convolutional graph attention deep neural network with multi-wind farm data.Based on the graph attention network and attention mechanism,the method extracts spatial-temporal characteristics from the data of multiple wind farms.Then,combined with a deep neural network,a convolutional graph attention deep neural network model is constructed.Finally,the model is trained with the quantile regression loss function to achieve the wind power deterministic and probabilistic prediction based on multi-wind farm spatial-temporal data.A wind power dataset in the U.S.is taken as an example to demonstrate the efficacy of the proposed model.Compared with the selected baseline methods,the proposed model achieves the best prediction performance.The point prediction errors(i.e.,root mean square error(RMSE)and normalized mean absolute percentage error(NMAPE))are 0.304 MW and 1.177%,respectively.And the comprehensive performance of probabilistic prediction(i.e.,con-tinuously ranked probability score(CRPS))is 0.580.Thus,the significance of multi-wind farm data and spatial-temporal feature extraction module is self-evident.

Electroencephalogram Signal Correlations between Default Mode Network and Attentional Functioning

Attentional issues may affect acquiring new information, task performance, and learning. Cortical network activities change during different functional brain states, including the default mode network (DMN) and attention network. We investigated the neural mechanisms underlying attentional functions and correlations between DMN connectivity and attentional function using the Trail-Making Test (TMT)-A and -B. Electroencephalography recordings were performed by placing 19 scalp electrodes per the 10 - 20 system. The mean power level was calculated for each rest and task condition. Non-parametric Spearman’s rank correlation was used to examine the correlation in power levels between the rest and TMT conditions. The most significant correlations during TMT-A were observed in the high gamma wave, followed by theta and beta waves, indicating that most correlations were in the parietal lobe, followed by the frontal, central, and temporal lobes. The most significant correlations during TMT-B were observed in the beta wave, followed by the high and low gamma waves, indicating that most correlations were in the temporal lobe, followed by the parietal, frontal, and central lobes. Frontoparietal beta and gamma waves in the DMN may represent attentional functions.

DHSEGATs:distance and hop-wise structures encoding enhanced graph attention networks

Numerous works prove that existing neighbor-averaging graph neural networks(GNNs)cannot efficiently catch structure features,and many works show that injecting structure,distance,position,or spatial features can significantly improve the performance of GNNs,however,injecting high-level structure and distance into GNNs is an intuitive but untouched idea.This work sheds light on this issue and proposes a scheme to enhance graph attention networks(GATs)by encoding distance and hop-wise structure statistics.Firstly,the hop-wise structure and distributional distance information are extracted based on several hop-wise ego-nets of every target node.Secondly,the derived structure information,distance information,and intrinsic features are encoded into the same vector space and then added together to get initial embedding vectors.Thirdly,the derived embedding vectors are fed into GATs,such as GAT and adaptive graph diffusion network(AGDN)to get the soft labels.Fourthly,the soft labels are fed into correct and smooth(C&S)to conduct label propagation and get final predictions.Experiments show that the distance and hop-wise structures encoding enhanced graph attention networks(DHSEGATs)achieve a competitive result.

Combing Type-Aware Attention and Graph Convolutional Networks for Event Detection

Event detection(ED)is aimed at detecting event occurrences and categorizing them.This task has been previously solved via recognition and classification of event triggers(ETs),which are defined as the phrase or word most clearly expressing event occurrence.Thus,current approaches require both annotated triggers as well as event types in training data.Nevertheless,triggers are non-essential in ED,and it is time-wasting for annotators to identify the“most clearly”word from a sentence,particularly in longer sentences.To decrease manual effort,we evaluate event detectionwithout triggers.We propose a novel framework that combines Type-aware Attention and Graph Convolutional Networks(TA-GCN)for event detection.Specifically,the task is identified as a multi-label classification problem.We first encode the input sentence using a novel type-aware neural network with attention mechanisms.Then,a Graph Convolutional Networks(GCN)-based multilabel classification model is exploited for event detection.Experimental results demonstrate the effectiveness.

Topic-Aware Abstractive Summarization Based on Heterogeneous Graph Attention Networks for Chinese Complaint Reports

Automatic text summarization(ATS)plays a significant role in Natural Language Processing(NLP).Abstractive summarization produces summaries by identifying and compressing the most important information in a document.However,there are only relatively several comprehensively evaluated abstractive summarization models that work well for specific types of reports due to their unstructured and oral language text characteristics.In particular,Chinese complaint reports,generated by urban complainers and collected by government employees,describe existing resident problems in daily life.Meanwhile,the reflected problems are required to respond speedily.Therefore,automatic summarization tasks for these reports have been developed.However,similar to traditional summarization models,the generated summaries still exist problems of informativeness and conciseness.To address these issues and generate suitably informative and less redundant summaries,a topic-based abstractive summarization method is proposed to obtain global and local features.Additionally,a heterogeneous graph of the original document is constructed using word-level and topic-level features.Experiments and analyses on public review datasets(Yelp and Amazon)and our constructed dataset(Chinese complaint reports)show that the proposed framework effectively improves the performance of the abstractive summarization model for Chinese complaint reports.

融合RoBERTa-GCN-Attention的隐喻识别与情感分类模型

在隐喻识别与隐喻情感分类任务的联合研究中,现有多任务学习模型存在对隐喻语料中的上下文语义信息和句法结构信息提取不够准确,并且缺乏对粗细两种粒度信息同时捕捉的问题.针对第1个问题,首先改进了传统的RoBERTa模型,在原有的自注意力机制中引入上下文信息,以此提取上下文中重要的隐喻语义特征;其次在句法依存树上使用图卷积网络提取隐喻句中的句法结构信息.针对第2个问题,使用双层注意力机制,分别聚焦于单词和句子层面中对隐喻识别和情感分类有贡献的特征信息.在两类任务6个数据集上的对比实验结果表明,该模型相比基线模型性能均有提升.

基于CNN-BiGRU-Attention的短期电力负荷预测

针对目前电力负荷数据随机性强,影响因素复杂,传统单一预测模型精度低的问题,结合卷积神经网络(Convolutional neural network,CNN)、双向门控循环单元(Bi-directional gated recurrent unit,BiGRU)以及注意力机制(Attention)在短期电力负荷预测上的不同优点,提出一种基于CNN-BiGRU-Attention的混合预测模型。该方法首先通过CNN对历史负荷和气象数据进行初步特征提取,然后利用BiGRU进一步挖掘特征数据间时序关联,再引入注意力机制,对BiGRU输出状态给与不同权重,强化关键特征,最后完成负荷预测。试验结果表明,该模型的平均绝对百分比误差(Mean absolute percentage error,MAPE)、均方根误差(Root mean square error,RMSE)、判定系数(R-square,R~2)分别为0.167%、0.057%、0.993,三项指标明显优于其他模型,具有更高的预测精度和稳定性,验证了模型在短期负荷预测中的优势。

基于CNN-LSTM-Attention的工业控制系统网络入侵检测方法研究

随着各类网络攻击事件的增加,能源电力基础设施中工业控制系统安全问题也逐渐成为人们关注的焦点。结合电力系统的特点,提出一种融合卷积神经网络(convolutional neural network,CNN)、长短时记忆(long short-term memory,LSTM)神经网络和注意力(Attention)机制的CNN-LSTM-Attention网络入侵检测算法模型,通过在实验室仿真环境中构造和采集600 MW燃煤机组制粉系统在3种典型工况下受到网络攻击的运行状态数据集,对所提出的检测算法模型进行训练和评估。结果表明:相较于CNN、LSTM模型,所提出的入侵检测算法模型性能最优;模型准确率、精确率、召回率等评级指标均为最好,综合评价优于其他的入侵检测方法。该入侵检测算法模型具有较强的创新性和实用性。

基于BiLSTM-Attention的F_(10.7)指数预测模型与中国自主数据集的应用

F_(10.7)指数是太阳活动的重要指标,准确预测F_(10.7)指数有助于预防和缓解太阳活动对无线电通信、导航和卫星通信等领域的影响.基于F_(10.7)射电流量的特性,在双向长短时记忆网络(Bidirectional Long Short-Term Memory Network,BiLSTM)基础上融入注意力机制(Attention),提出了一种基于BiLSTM-Attention的F_(10.7)预报模型.在加拿大DRAO数据集上其平均绝对误差(MAE)为5.38,平均绝对百分比误差(MAPE)控制在5%以内,相关系数(R)高达0.987,与其他RNN模型相比拥有优越的预测性能.针对中国廊坊L&S望远镜观测的F_(10.7)数据集,提出了一种转换平均校准(Conversion Average Calibration,CAC)方法进行数据预处理,处理后的数据与DRAO数据集具有较高的相关性.基于该数据集对比分析了RNN系列模型的预报效果,实验结果表明,BiLSTM-Attention和BiLSTM两种模型在预测F_(10.7)指数方面具有较好的优势,表现出较好的预测性能和稳定性.

基于SSA-CG-Attention模型的多因素采煤工作面涌水量预测

矿井工作面涌水量预测对确保矿山安全、优化资源配置、提高工作效率等都具有重要作用。为提高预测结果的准确性和稳定性,基于钻孔水位和微震能量数据与涌水量的强关联性,选择其作为多因素特征变量,提出SSA-CG-Attention多因素矿井工作面涌水量预测模型。该模型在门控循环单元(GatedRecurrentUnit,GRU)提取时序特征的基础上,与卷积神经网络(ConvolutionalNeuralNet-work,CNN)融合形成新的网络结构提取数据的有效非线性局部特征,并且加入注意力机制(Atten-tion),在预测过程中将注意力集中在输入元素上,提高模型的准确性。最后通过麻雀搜索算法(Spar-row Search Algorithm,SSA)优化模型参数,避免局部最优解的问题。将提出的模型分别与传统的BP神经网络、LSTM、GRU单因素涌水量预测模型以及MLP、SLP、SVR、LSTM、GRU、SSA-LSTM、SSA-GRU多因素涌水量预测模型的预测结果进行对比分析,结果表明:SSA算法以最少迭代次数快速寻优,避免了局部最优解的缺陷;SSA-CG-Attention多因素涌水量预测模型整体预测指标绝对误差(E_(MA))、均方根误差(E_(RMS))以及平均绝对百分比误差(E_(MAP))分别为5.24 m^(3)/h、7.25 m^(3)/h、6%,指标方差和为8.90。相较于其他预测模型预测精度更高,相较于单因素涌水量预测模型,多因素涌水量预测模型预测结果更加稳定。研究结果为矿井工作面涌水量预测提供了新的思路与方法,对矿井工作面涌水量预测及防控有着借鉴与指导作用,具有一定的理论价值和现实意义。

基于Attention-BP神经网络模型的邮轮客舱火灾危险等级分类研究

为能够对邮轮客舱不同火灾危险源进行风险评估,提出一种可对舱室火灾危险等级实时分类的新型神经网络模型。通过火灾动态模拟器(FDS)建立邮轮客舱火灾物理模型,对发生火灾时的烟气温度、CO体积分数和能见度等安全指标进行数值模拟,并基于其对人体的影响程度将火灾危险等级划分为4个等级。通过设计一种新型的Attention-BP神经网络(BPNN)模型,结合self-Attention机制融合多个BPNN诊断结果,自适应地分配各个BPNN的权重,对采集的多源火灾信息进行分析和处理,实现对客舱火灾的风险评估并划分危险等级。试验证明:Attention-BPNN模型可有效地实现对火灾危险等级的预警,准确率可达97.32%。相对于其他机器学习算法,具有最高的稳定性和准确率,减少了对客舱火灾预警的不确定性。

Predicting Traffic Flow Using Dynamic Spatial-Temporal Graph Convolution Networks

Traffic flow prediction plays a key role in the construction of intelligent transportation system.However,due to its complex spatio-temporal dependence and its uncertainty,the research becomes very challenging.Most of the existing studies are based on graph neural networks that model traffic flow graphs and try to use fixed graph structure to deal with the relationship between nodes.However,due to the time-varying spatial correlation of the traffic network,there is no fixed node relationship,and these methods cannot effectively integrate the temporal and spatial features.This paper proposes a novel temporal-spatial dynamic graph convolutional network(TSADGCN).The dynamic time warping algorithm(DTW)is introduced to calculate the similarity of traffic flow sequence among network nodes in the time dimension,and the spatiotemporal graph of traffic flow is constructed to capture the spatiotemporal characteristics and dependencies of traffic flow.By combining graph attention network and time attention network,a spatiotemporal convolution block is constructed to capture spatiotemporal characteristics of traffic data.Experiments on open data sets PEMSD4 and PEMSD8 show that TSADGCN has higher prediction accuracy than well-known traffic flow prediction algorithms.

Lightweight Cross-Modal Multispectral Pedestrian Detection Based on Spatial Reweighted Attention Mechanism

Multispectral pedestrian detection technology leverages infrared images to provide reliable information for visible light images, demonstrating significant advantages in low-light conditions and background occlusion scenarios. However, while continuously improving cross-modal feature extraction and fusion, ensuring the model’s detection speed is also a challenging issue. We have devised a deep learning network model for cross-modal pedestrian detection based on Resnet50, aiming to focus on more reliable features and enhance the model’s detection efficiency. This model employs a spatial attention mechanism to reweight the input visible light and infrared image data, enhancing the model’s focus on different spatial positions and sharing the weighted feature data across different modalities, thereby reducing the interference of multi-modal features. Subsequently, lightweight modules with depthwise separable convolution are incorporated to reduce the model’s parameter count and computational load through channel-wise and point-wise convolutions. The network model algorithm proposed in this paper was experimentally validated on the publicly available KAIST dataset and compared with other existing methods. The experimental results demonstrate that our approach achieves favorable performance in various complex environments, affirming the effectiveness of the multispectral pedestrian detection technology proposed in this paper.

Social Robot Detection Method with Improved Graph Neural Networks

Social robot accounts controlled by artificial intelligence or humans are active in social networks,bringing negative impacts to network security and social life.Existing social robot detection methods based on graph neural networks suffer from the problem of many social network nodes and complex relationships,which makes it difficult to accurately describe the difference between the topological relations of nodes,resulting in low detection accuracy of social robots.This paper proposes a social robot detection method with the use of an improved neural network.First,social relationship subgraphs are constructed by leveraging the user’s social network to disentangle intricate social relationships effectively.Then,a linear modulated graph attention residual network model is devised to extract the node and network topology features of the social relation subgraph,thereby generating comprehensive social relation subgraph features,and the feature-wise linear modulation module of the model can better learn the differences between the nodes.Next,user text content and behavioral gene sequences are extracted to construct social behavioral features combined with the social relationship subgraph features.Finally,social robots can be more accurately identified by combining user behavioral and relationship features.By carrying out experimental studies based on the publicly available datasets TwiBot-20 and Cresci-15,the suggested method’s detection accuracies can achieve 86.73%and 97.86%,respectively.Compared with the existing mainstream approaches,the accuracy of the proposed method is 2.2%and 1.35%higher on the two datasets.The results show that the method proposed in this paper can effectively detect social robots and maintain a healthy ecological environment of social networks.

融合MacBERT和Talking⁃Heads Attention实体关系联合抽取模型

针对现有的医学文本关系抽取任务模型在训练过程中存在语义理解能力不足,可能导致关系抽取的效果不尽人意的问题,文中提出一种融合MacBERT和Talking⁃Heads Attention的实体关系联合抽取模型。该模型首先利用MacBERT语言模型来获取动态字向量表达,MacBERT作为改进的BERT模型,能够减少预训练和微调阶段之间的差异,从而提高模型的泛化能力;然后,将这些动态字向量表达输入到双向门控循环单元(BiGRU)中,以便提取文本的上下文特征。BiGRU是一种改进的循环神经网络(RNN),具有更好的长期依赖捕获能力。在获取文本上下文特征之后,使用Talking⁃Heads Attention来获取全局特征。Talking⁃Heads Attention是一种自注意力机制,可以捕获文本中不同位置之间的关系,从而提高关系抽取的准确性。实验结果表明,与实体关系联合抽取模型GRTE相比,该模型F1值提升1%,precision值提升0.4%,recall值提升1.5%。

Continuous Sign Language Recognition Based on Spatial-Temporal Graph Attention Network

Continuous sign language recognition(CSLR)is challenging due to the complexity of video background,hand gesture variability,and temporal modeling difficulties.This work proposes a CSLR method based on a spatialtemporal graph attention network to focus on essential features of video series.The method considers local details of sign language movements by taking the information on joints and bones as inputs and constructing a spatialtemporal graph to reflect inter-frame relevance and physical connections between nodes.The graph-based multihead attention mechanism is utilized with adjacent matrix calculation for better local-feature exploration,and short-term motion correlation modeling is completed via a temporal convolutional network.We adopted BLSTM to learn the long-termdependence and connectionist temporal classification to align the word-level sequences.The proposed method achieves competitive results regarding word error rates(1.59%)on the Chinese Sign Language dataset and the mean Jaccard Index(65.78%)on the ChaLearn LAP Continuous Gesture Dataset.

Short-Term Household Load Forecasting Based on Attention Mechanism and CNN-ICPSO-LSTM

Accurate load forecasting forms a crucial foundation for implementing household demand response plans andoptimizing load scheduling. When dealing with short-term load data characterized by substantial fluctuations,a single prediction model is hard to capture temporal features effectively, resulting in diminished predictionaccuracy. In this study, a hybrid deep learning framework that integrates attention mechanism, convolution neuralnetwork (CNN), improved chaotic particle swarm optimization (ICPSO), and long short-term memory (LSTM), isproposed for short-term household load forecasting. Firstly, the CNN model is employed to extract features fromthe original data, enhancing the quality of data features. Subsequently, the moving average method is used for datapreprocessing, followed by the application of the LSTM network to predict the processed data. Moreover, the ICPSOalgorithm is introduced to optimize the parameters of LSTM, aimed at boosting the model’s running speed andaccuracy. Finally, the attention mechanism is employed to optimize the output value of LSTM, effectively addressinginformation loss in LSTM induced by lengthy sequences and further elevating prediction accuracy. According tothe numerical analysis, the accuracy and effectiveness of the proposed hybrid model have been verified. It canexplore data features adeptly, achieving superior prediction accuracy compared to other forecasting methods forthe household load exhibiting significant fluctuations across different seasons.

A novel intrusion detection model for the CAN bus packet of in-vehicle network based on attention mechanism and autoencoder

The attacks on in-vehicle Controller Area Network(CAN)bus messages severely disrupt normal communication between vehicles.Therefore,researches on intrusion detection models for CAN have positive business value for vehicle security,and the intrusion detection technology for CAN bus messages can effectively protect the invehicle network from unlawful attacks.Previous machine learning-based models are unable to effectively identify intrusive abnormal messages due to their inherent shortcomings.Hence,to address the shortcomings of the previous machine learning-based intrusion detection technique,we propose a novel method using Attention Mechanism and AutoEncoder for Intrusion Detection(AMAEID).The AMAEID model first converts the raw hexadecimal message data into binary format to obtain better input.Then the AMAEID model encodes and decodes the binary message data using a multi-layer denoising autoencoder model to obtain a hidden feature representation that can represent the potential features behind the message data at a deeper level.Finally,the AMAEID model uses the attention mechanism and the fully connected layer network to infer whether the message is an abnormal message or not.The experimental results with three evaluation metrics on a real in-vehicle CAN bus message dataset outperform some traditional machine learning algorithms,demonstrating the effectiveness of the AMAEID model.