Clothing Parsing Based on Multi-Scale Fusion and Improved Self-Attention Mechanism

Due to the lack of long-range association and spatial location information,fine details and accurate boundaries of complex clothing images cannot always be obtained by using the existing deep learning-based methods.This paper presents a convolutional structure with multi-scale fusion to optimize the step of clothing feature extraction and a self-attention module to capture long-range association information.The structure enables the self-attention mechanism to directly participate in the process of information exchange through the down-scaling projection operation of the multi-scale framework.In addition,the improved self-attention module introduces the extraction of 2-dimensional relative position information to make up for its lack of ability to extract spatial position features from clothing images.The experimental results based on the colorful fashion parsing dataset(CFPD)show that the proposed network structure achieves 53.68%mean intersection over union(mIoU)and has better performance on the clothing parsing task.

Hierarchical multihead self-attention for time-series-based fault diagnosis

Fault diagnosis is important for maintaining the safety and effectiveness of chemical process.Considering the multivariate,nonlinear,and dynamic characteristic of chemical process,many time-series-based data-driven fault diagnosis methods have been developed in recent years.However,the existing methods have the problem of long-term dependency and are difficult to train due to the sequential way of training.To overcome these problems,a novel fault diagnosis method based on time-series and the hierarchical multihead self-attention(HMSAN)is proposed for chemical process.First,a sliding window strategy is adopted to construct the normalized time-series dataset.Second,the HMSAN is developed to extract the time-relevant features from the time-series process data.It improves the basic self-attention model in both width and depth.With the multihead structure,the HMSAN can pay attention to different aspects of the complicated chemical process and obtain the global dynamic features.However,the multiple heads in parallel lead to redundant information,which cannot improve the diagnosis performance.With the hierarchical structure,the redundant information is reduced and the deep local time-related features are further extracted.Besides,a novel many-to-one training strategy is introduced for HMSAN to simplify the training procedure and capture the long-term dependency.Finally,the effectiveness of the proposed method is demonstrated by two chemical cases.The experimental results show that the proposed method achieves a great performance on time-series industrial data and outperforms the state-of-the-art approaches.

Self-Attention Spatio-Temporal Deep Collaborative Network for Robust FDIA Detection in Smart Grids

False data injection attack(FDIA)can affect the state estimation of the power grid by tampering with the measured value of the power grid data,and then destroying the stable operation of the smart grid.Existing work usually trains a detection model by fusing the data-driven features from diverse power data streams.Data-driven features,however,cannot effectively capture the differences between noisy data and attack samples.As a result,slight noise disturbances in the power grid may cause a large number of false detections for FDIA attacks.To address this problem,this paper designs a deep collaborative self-attention network to achieve robust FDIA detection,in which the spatio-temporal features of cascaded FDIA attacks are fully integrated.Firstly,a high-order Chebyshev polynomials-based graph convolution module is designed to effectively aggregate the spatio information between grid nodes,and the spatial self-attention mechanism is involved to dynamically assign attention weights to each node,which guides the network to pay more attention to the node information that is conducive to FDIA detection.Furthermore,the bi-directional Long Short-Term Memory(LSTM)network is introduced to conduct time series modeling and long-term dependence analysis for power grid data and utilizes the temporal selfattention mechanism to describe the time correlation of data and assign different weights to different time steps.Our designed deep collaborative network can effectively mine subtle perturbations from spatiotemporal feature information,efficiently distinguish power grid noise from FDIA attacks,and adapt to diverse attack intensities.Extensive experiments demonstrate that our method can obtain an efficient detection performance over actual load data from New York Independent System Operator(NYISO)in IEEE 14,IEEE 39,and IEEE 118 bus systems,and outperforms state-of-the-art FDIA detection schemes in terms of detection accuracy and robustness.

Sentiment classification model for bullet screen based on self-attention mechanism

With the development of short video industry,video and bullet screen have become important ways to spread public opinions.Public attitudes can be timely obtained through emotional analysis on bullet screen,which can also reduce difficulties in management of online public opinions.A convolutional neural network model based on multi-head attention is proposed to solve the problem of how to effectively model relations among words and identify key words in emotion classification tasks with short text contents and lack of complete context information.Firstly,encode word positions so that order information of input sequences can be used by the model.Secondly,use a multi-head attention mechanism to obtain semantic expressions in different subspaces,effectively capture internal relevance and enhance dependent relationships among words,as well as highlight emotional weights of key emotional words.Then a dilated convolution is used to increase the receptive field and extract more features.On this basis,the above multi-attention mechanism is combined with a convolutional neural network to model and analyze the seven emotional categories of bullet screens.Testing from perspectives of model and dataset,experimental results can validate effectiveness of our approach.Finally,emotions of bullet screens are visualized to provide data supports for hot event controls and other fields.

Keyphrase Generation Based on Self-Attention Mechanism

Keyphrase greatly provides summarized and valuable information.This information can help us not only understand text semantics,but also organize and retrieve text content effectively.The task of automatically generating it has received considerable attention in recent decades.From the previous studies,we can see many workable solutions for obtaining keyphrases.One method is to divide the content to be summarized into multiple blocks of text,then we rank and select the most important content.The disadvantage of this method is that it cannot identify keyphrase that does not include in the text,let alone get the real semantic meaning hidden in the text.Another approach uses recurrent neural networks to generate keyphrases from the semantic aspects of the text,but the inherently sequential nature precludes parallelization within training examples,and distances have limitations on context dependencies.Previous works have demonstrated the benefits of the self-attention mechanism,which can learn global text dependency features and can be parallelized.Inspired by the above observation,we propose a keyphrase generation model,which is based entirely on the self-attention mechanism.It is an encoder-decoder model that can make up the above disadvantage effectively.In addition,we also consider the semantic similarity between keyphrases,and add semantic similarity processing module into the model.This proposed model,which is demonstrated by empirical analysis on five datasets,can achieve competitive performance compared to baseline methods.

NFHP-RN:AMethod of Few-Shot Network Attack Detection Based on the Network Flow Holographic Picture-ResNet

Due to the rapid evolution of Advanced Persistent Threats(APTs)attacks,the emergence of new and rare attack samples,and even those never seen before,make it challenging for traditional rule-based detection methods to extract universal rules for effective detection.With the progress in techniques such as transfer learning and meta-learning,few-shot network attack detection has progressed.However,challenges in few-shot network attack detection arise from the inability of time sequence flow features to adapt to the fixed length input requirement of deep learning,difficulties in capturing rich information from original flow in the case of insufficient samples,and the challenge of high-level abstract representation.To address these challenges,a few-shot network attack detection based on NFHP(Network Flow Holographic Picture)-RN(ResNet)is proposed.Specifically,leveraging inherent properties of images such as translation invariance,rotation invariance,scale invariance,and illumination invariance,network attack traffic features and contextual relationships are intuitively represented in NFHP.In addition,an improved RN network model is employed for high-level abstract feature extraction,ensuring that the extracted high-level abstract features maintain the detailed characteristics of the original traffic behavior,regardless of changes in background traffic.Finally,a meta-learning model based on the self-attention mechanism is constructed,achieving the detection of novel APT few-shot network attacks through the empirical generalization of high-level abstract feature representations of known-class network attack behaviors.Experimental results demonstrate that the proposed method can learn high-level abstract features of network attacks across different traffic detail granularities.Comparedwith state-of-the-artmethods,it achieves favorable accuracy,precision,recall,and F1 scores for the identification of unknown-class network attacks through cross-validation onmultiple datasets.

A New Industrial Intrusion Detection Method Based on CNN-BiLSTM

Nowadays,with the rapid development of industrial Internet technology,on the one hand,advanced industrial control systems(ICS)have improved industrial production efficiency.However,there are more and more cyber-attacks targeting industrial control systems.To ensure the security of industrial networks,intrusion detection systems have been widely used in industrial control systems,and deep neural networks have always been an effective method for identifying cyber attacks.Current intrusion detection methods still suffer from low accuracy and a high false alarm rate.Therefore,it is important to build a more efficient intrusion detection model.This paper proposes a hybrid deep learning intrusion detection method based on convolutional neural networks and bidirectional long short-term memory neural networks(CNN-BiLSTM).To address the issue of imbalanced data within the dataset and improve the model’s detection capabilities,the Synthetic Minority Over-sampling Technique-Edited Nearest Neighbors(SMOTE-ENN)algorithm is applied in the preprocessing phase.This algorithm is employed to generate synthetic instances for the minority class,simultaneously mitigating the impact of noise in the majority class.This approach aims to create a more equitable distribution of classes,thereby enhancing the model’s ability to effectively identify patterns in both minority and majority classes.In the experimental phase,the detection performance of the method is verified using two data sets.Experimental results show that the accuracy rate on the CICIDS-2017 data set reaches 97.7%.On the natural gas pipeline dataset collected by Lan Turnipseed from Mississippi State University in the United States,the accuracy rate also reaches 85.5%.

Intelligent Fault Diagnosis Method of Rolling Bearings Based on Transfer Residual Swin Transformer with Shifted Windows

Due to their robust learning and expression ability for complex features,the deep learning(DL)model plays a vital role in bearing fault diagnosis.However,since there are fewer labeled samples in fault diagnosis,the depth of DL models in fault diagnosis is generally shallower than that of DL models in other fields,which limits the diagnostic performance.To solve this problem,a novel transfer residual Swin Transformer(RST)is proposed for rolling bearings in this paper.RST has 24 residual self-attention layers,which use the hierarchical design and the shifted window-based residual self-attention.Combined with transfer learning techniques,the transfer RST model uses pre-trained parameters from ImageNet.A new end-to-end method for fault diagnosis based on deep transfer RST is proposed.Firstly,wavelet transform transforms the vibration signal into a wavelet time-frequency diagram.The signal’s time-frequency domain representation can be represented simultaneously.Secondly,the wavelet time-frequency diagram is the input of the RST model to obtain the fault type.Finally,our method is verified on public and self-built datasets.Experimental results show the superior performance of our method by comparing it with a shallow neural network.

基于概率化稀疏自注意力LSTM的锂离子电池健康状态预测

针对锂离子电池健康状态(state of health,SOH)预测,提出了一种基于概率化稀疏自注意力机制(probsparseself-attentionmechanism,PSM)和长短期记忆(longshort-term memory,LSTM)神经网络的预测模型。首先,提取锂离子电池容量数据并进行窗口化处理,利用位置嵌入获取高维数据之间的特征信息并对数据进行位置编码。然后,引入PSM对输入数据的权重进行稀疏性判断,增加对SOH预测具有关键影响的因素的权重。最后,利用LSTM神经网络捕获数据之间的时序特征进行锂离子电池SOH预测。实验结果表明,与其他常用的锂离子电池SOH预测模型相比,所提模型可以减少预测误差,具有更好的预测性能。

改进Informer模型的苜蓿土壤湿度预测方法

精准的苜蓿土壤湿度预测对于提高水资源利用率和降低智慧农业投入成本至关重要。针对传统土壤湿度预测方法在实际应用中存在预测周期短、精度低以及时空预测不足等问题,提出了一种融合快速傅里叶变换的Informer时空预测方法(Fast Fourier Transform and Spatio Temporal-Informer,FFT-ST-Informer)。首先,在传统Informer模型基础上添加了独立的时空嵌入层,从而捕获各个变量之间复杂的时空相关性。然后,根据土壤墒情与环境因素的相关性分析结果,选择降雨、灌溉量为关键环境因素,并使用快速傅里叶变换,通过提取某一周期具有先验的数据序列的频谱来表示其频域特征放入模型。此外,该模型中的ProbSparse自注意机制可以集中提取时空数据的重要上下文信息。FFT-ST-Informer模型使用来自宁夏引黄灌区自采的气象和土壤数据作为输入数据。实验结果表明,FFT-ST-Informer模型性能明显优于传统模型,比LSTM模型在平均绝对误差(MAE)、均方根误差(RMSE)、相关系数(R^(2))等评价指标上,分别提高了56.9%,64.4%,0.12%。

An Affective EEG Analysis Method Without Feature Engineering

Emotional electroencephalography(EEG)signals are a primary means of recording emotional brain activity.Currently,the most effective methods for analyzing emotional EEG signals involve feature engineering and neural networks.However,neural networks possess a strong ability for automatic feature extraction.Is it possible to discard feature engineering and directly employ neural networks for end-to-end recognition?Based on the characteristics of EEG signals,this paper proposes an end-to-end feature extraction and classification method for a dynamic self-attention network(DySAT).The study reveals significant differences in brain activity patterns associated with different emotions across various experimenters and time periods.The results of this experiment can provide insights into the reasons behind these differences.

circ2CBA: prediction of circRNA-RBP binding sites combining deep learning and attention mechanism

Circular RNAs(circRNAs)are RNAs with closed circular structure involved in many biological processes by key interactions with RNA binding proteins(RBPs).Existing methods for predicting these interactions have limitations in feature learning.In view of this,we propose a method named circ2CBA,which uses only sequence information of circRNAs to predict circRNA-RBP binding sites.We have constructed a data set which includes eight sub-datasets.First,circ2CBA encodes circRNA sequences using the one-hot method.Next,a two-layer convolutional neural network(CNN)is used to initially extract the features.After CNN,circ2CBA uses a layer of bidirectional long and short-term memory network(BiLSTM)and the self-attention mechanism to learn the features.The AUC value of circ2CBA reaches 0.8987.Comparison of circ2CBA with other three methods on our data set and an ablation experiment confirm that circ2CBA is an effective method to predict the binding sites between circRNAs and RBPs.

3D Object Detection with Attention:Shell-Based Modeling

LIDAR point cloud-based 3D object detection aims to sense the surrounding environment by anchoring objects with the Bounding Box(BBox).However,under the three-dimensional space of autonomous driving scenes,the previous object detection methods,due to the pre-processing of the original LIDAR point cloud into voxels or pillars,lose the coordinate information of the original point cloud,slow detection speed,and gain inaccurate bounding box positioning.To address the issues above,this study proposes a new two-stage network structure to extract point cloud features directly by PointNet++,which effectively preserves the original point cloud coordinate information.To improve the detection accuracy,a shell-based modeling method is proposed.It roughly determines which spherical shell the coordinates belong to.Then,the results are refined to ground truth,thereby narrowing the localization range and improving the detection accuracy.To improve the recall of 3D object detection with bounding boxes,this paper designs a self-attention module for 3D object detection with a skip connection structure.Some of these features are highlighted by weighting them on the feature dimensions.After training,it makes the feature weights that are favorable for object detection get larger.Thus,the extracted features are more adapted to the object detection task.Extensive comparison experiments and ablation experiments conducted on the KITTI dataset verify the effectiveness of our proposed method in improving recall and precision.

Research on Multi-Modal Time Series Data Prediction Method Based on Dual-Stage Attention Mechanism

The production data in the industrialfield have the characteristics of multimodality,high dimensionality and large correlation differences between attributes.Existing data prediction methods cannot effectively capture time series and modal features,which leads to prediction hysteresis and poor prediction stabil-ity.Aiming at the above problems,this paper proposes a time-series and modal fea-tureenhancementmethodbasedonadual-stageself-attentionmechanism(DATT),and a time series prediction method based on a gated feedforward recurrent unit(GFRU).On this basis,the DATT-GFRU neural network with a gated feedforward recurrent neural network and dual-stage self-attention mechanism is designed and implemented.Experiments show that the prediction effect of the neural network prediction model based on DATT is significantly improved.Compared with the traditional prediction model,the DATT-GFRU neural network has a smaller aver-age error of model prediction results,stable prediction performance,and strong generalization ability on the three datasets with different numbers of attributes and different training sample sizes.

Multi-head attention-based long short-term memory model for speech emotion recognition

To fully make use of information from different representation subspaces,a multi-head attention-based long short-term memory(LSTM)model is proposed in this study for speech emotion recognition(SER).The proposed model uses frame-level features and takes the temporal information of emotion speech as the input of the LSTM layer.Here,a multi-head time-dimension attention(MHTA)layer was employed to linearly project the output of the LSTM layer into different subspaces for the reduced-dimension context vectors.To provide relative vital information from other dimensions,the output of MHTA,the output of feature-dimension attention,and the last time-step output of LSTM were utilized to form multiple context vectors as the input of the fully connected layer.To improve the performance of multiple vectors,feature-dimension attention was employed for the all-time output of the first LSTM layer.The proposed model was evaluated on the eNTERFACE and GEMEP corpora,respectively.The results indicate that the proposed model outperforms LSTM by 14.6%and 10.5%for eNTERFACE and GEMEP,respectively,proving the effectiveness of the proposed model in SER tasks.

Saliency guided self-attention network for pedestrian attribute recognition in surveillance scenarios

Pedestrian attribute recognition is often considered as a multi-label image classification task. In order to make full use of attribute-related location information, a saliency guided self-attention network(SGSA-Net) was proposed to weakly supervise attribute localization, without annotations of attribute-related regions. Saliency priors were integrated into the spatial attention module(SAM). Meanwhile, channel-wise attention and spatial attention were introduced into the network. Moreover, a weighted binary cross-entropy loss(WCEL) function was employed to handle the imbalance of training data. Extensive experiments on richly annotated pedestrian(RAP) and pedestrian attribute(PETA) datasets demonstrated that SGSA-Net outperformed other state-of-the-art methods.

Research on clothing patterns generation based on multi-scales self-attention improved generative adversarial network

Purpose-Clothing patterns play a dominant role in costume design and have become an important link in the perception of costume art.Conventional clothing patterns design relies on experienced designers.Although the quality of clothing patterns is very high on conventional design,the input time and output amount ratio is relative low for conventional design.In order to break through the bottleneck of conventional clothing patterns design,this paper proposes a novel way based on generative adversarial network(GAN)model for automatic clothing patterns generation,which not only reduces the dependence of experienced designer,but also improve the input-output ratio.Design/methodology/approach-In view of the fact that clothing patterns have high requirements for global artistic perception and local texture details,this paper improves the conventional GAN model from two aspects:a multi-scales discriminators strategy is introduced to deal with the local texture details;and the selfattention mechanism is introduced to improve the global artistic perception.Therefore,the improved GAN called multi-scales self-attention improved generative adversarial network(MS-SA-GAN)model,which is used for high resolution clothing patterns generation.Findings-To verify the feasibility and effectiveness of the proposed MS-SA-GAN model,a crawler is designed to acquire standard clothing patterns dataset from Baidu pictures,and a comparative experiment is conducted on our designed clothing patterns dataset.In experiments,we have adjusted different parameters of the proposed MS-SA-GAN model,and compared the global artistic perception and local texture details of the generated clothing patterns.Originality/value-Experimental results have shown that the clothing patterns generated by the proposed MS-SA-GANmodel are superior to the conventional algorithms in some local texture detail indexes.In addition,a group of clothing design professionals is invited to evaluate the global artistic perception through a valencearousal scale.The scale results have shown that the proposed MS-SA-GAN model achieves a better global art perception.

Self-attention Based Multimodule Fusion Graph Convolution Network for Traffic Flow Prediction

With rapid economic development,the per capita ownership of automobiles in our country has begun to rise year by year.More researchers have paid attention to using scientific methods to solve traffic flow problems.Traffic flow prediction is not simply affected by the number of vehicles,but also contains various complex factors,such as time,road conditions,and people flow.However,the existing methods ignore the complexity of road conditions and the correlation between individual nodes,which leads to the poor performance.In this study,a deep learning model SAMGCN is proposed to effectively capture the correlation between individual nodes to improve the performance of traffic flow prediction.First,the theory of spatiotemporal decoupling is used to divide each time of each node into finer particles.Second,multimodule fusion is used to mine the potential periodic relationships in the data.Finally,GRU is used to obtain the potential time relationship of the three modules.Extensive experiments were conducted on two traffic flow datasets,PeMS04 and PeMS08 in the Caltrans Performance Measurement System to prove the validity of the proposed model.

基于概率稀疏自注意力的船舶短期电力负荷预测

针对船舶电力负荷数据预测时存在的实时性差、存储数据量小且质量低等问题,提出一种结合数据插补、小波阈值去噪与概率稀疏自注意力(ProbSparse self⁃atten⁃tion)机制的新型短期负荷预测方法。首先,在数据预处理阶段,在不影响原始数据特征及趋势前提下通过插值填补缺失数据,扩充数据库以满足模型训练要求,同时考虑到原始船舶电力负荷数据可能存在噪声干扰等问题,为减小其对模型预测效果的影响,对原信号采用了小波阈值去噪处理的方法来改善数据质量。其次,在预测模型中引入概率稀疏自注意力机制,在有效捕获时序电力数据中的依赖关系和重要特征的同时,降低内存资源占用,减小模型复杂度,满足船舶电力负荷预测实时性要求,实现了预测精度与效率双优化。相较传统Transformer模型、ARIMA和LSTM模型,本文模型在均方根误差和平均绝对百分比误差上平均分别降低了13.1%、18.6%,效率平均提高24.0%以上,表明本文方法在船舶电力负荷数据预测模型准确度及效率上具有明显优势。

TibetanGoTinyNet:a lightweight U-Net style network for zero learning of Tibetan Go

The game of Tibetan Go faces the scarcity of expert knowledge and research literature.Therefore,we study the zero learning model of Tibetan Go under limited computing power resources and propose a novel scaleinvariant U-Net style two-headed output lightweight network TibetanGoTinyNet.The lightweight convolutional neural networks and capsule structure are applied to the encoder and decoder of TibetanGoTinyNet to reduce computational burden and achieve better feature extraction results.Several autonomous self-attention mechanisms are integrated into TibetanGoTinyNet to capture the Tibetan Go board’s spatial and global information and select important channels.The training data are generated entirely from self-play games.TibetanGoTinyNet achieves 62%–78%winning rate against other four U-Net style models including Res-UNet,Res-UNet Attention,Ghost-UNet,and Ghost Capsule-UNet.It also achieves 75%winning rate in the ablation experiments on the attention mechanism with embedded positional information.The model saves about 33%of the training time with 45%–50%winning rate for different Monte–Carlo tree search(MCTS)simulation counts when migrated from 9×9 to 11×11 boards.Code for our model is available at https://github.com/paulzyy/TibetanGoTinyNet.