Hierarchical multihead self-attention for time-series-based fault diagnosis

Fault diagnosis is important for maintaining the safety and effectiveness of chemical process.Considering the multivariate,nonlinear,and dynamic characteristic of chemical process,many time-series-based data-driven fault diagnosis methods have been developed in recent years.However,the existing methods have the problem of long-term dependency and are difficult to train due to the sequential way of training.To overcome these problems,a novel fault diagnosis method based on time-series and the hierarchical multihead self-attention(HMSAN)is proposed for chemical process.First,a sliding window strategy is adopted to construct the normalized time-series dataset.Second,the HMSAN is developed to extract the time-relevant features from the time-series process data.It improves the basic self-attention model in both width and depth.With the multihead structure,the HMSAN can pay attention to different aspects of the complicated chemical process and obtain the global dynamic features.However,the multiple heads in parallel lead to redundant information,which cannot improve the diagnosis performance.With the hierarchical structure,the redundant information is reduced and the deep local time-related features are further extracted.Besides,a novel many-to-one training strategy is introduced for HMSAN to simplify the training procedure and capture the long-term dependency.Finally,the effectiveness of the proposed method is demonstrated by two chemical cases.The experimental results show that the proposed method achieves a great performance on time-series industrial data and outperforms the state-of-the-art approaches.

SMSTracker:A Self-Calibration Multi-Head Self-Attention Transformer for Visual Object Tracking

Visual object tracking plays a crucial role in computer vision.In recent years,researchers have proposed various methods to achieve high-performance object tracking.Among these,methods based on Transformers have become a research hotspot due to their ability to globally model and contextualize information.However,current Transformer-based object tracking methods still face challenges such as low tracking accuracy and the presence of redundant feature information.In this paper,we introduce self-calibration multi-head self-attention Transformer(SMSTracker)as a solution to these challenges.It employs a hybrid tensor decomposition self-organizing multihead self-attention transformermechanism,which not only compresses and accelerates Transformer operations but also significantly reduces redundant data,thereby enhancing the accuracy and efficiency of tracking.Additionally,we introduce a self-calibration attention fusion block to resolve common issues of attention ambiguities and inconsistencies found in traditional trackingmethods,ensuring the stability and reliability of tracking performance across various scenarios.By integrating a hybrid tensor decomposition approach with a self-organizingmulti-head self-attentive transformer mechanism,SMSTracker enhances the efficiency and accuracy of the tracking process.Experimental results show that SMSTracker achieves competitive performance in visual object tracking,promising more robust and efficient tracking systems,demonstrating its potential to providemore robust and efficient tracking solutions in real-world applications.

A Self-Attention Based Dynamic Resource Management for Satellite-Terrestrial Networks

The satellite-terrestrial networks possess the ability to transcend geographical constraints inherent in traditional communication networks,enabling global coverage and offering users ubiquitous computing power support,which is an important development direction of future communications.In this paper,we take into account a multi-scenario network model under the coverage of low earth orbit(LEO)satellite,which can provide computing resources to users in faraway areas to improve task processing efficiency.However,LEO satellites experience limitations in computing and communication resources and the channels are time-varying and complex,which makes the extraction of state information a daunting task.Therefore,we explore the dynamic resource management issue pertaining to joint computing,communication resource allocation and power control for multi-access edge computing(MEC).In order to tackle this formidable issue,we undertake the task of transforming the issue into a Markov decision process(MDP)problem and propose the self-attention based dynamic resource management(SABDRM)algorithm,which effectively extracts state information features to enhance the training process.Simulation results show that the proposed algorithm is capable of effectively reducing the long-term average delay and energy consumption of the tasks.

An Aerial Target Recognition Algorithm Based on Self-Attention and LSTM

In the application of aerial target recognition,on the one hand,the recognition error produced by the single measurement of the sensor is relatively large due to the impact of noise.On the other hand,it is difficult to apply machine learning methods to improve the intelligence and recognition effect due to few or no actual measurement samples.Aiming at these problems,an aerial target recognition algorithm based on self-attention and Long Short-Term Memory Network(LSTM)is proposed.LSTM can effectively extract temporal dependencies.The attention mechanism calculates the weight of each input element and applies the weight to the hidden state of the LSTM,thereby adjusting the LSTM’s attention to the input.This combination retains the learning ability of LSTM and introduces the advantages of the attention mechanism,making the model have stronger feature extraction ability and adaptability when processing sequence data.In addition,based on the prior information of the multidimensional characteristics of the target,the three-point estimation method is adopted to simulate an aerial target recognition dataset to train the recognition model.The experimental results show that the proposed algorithm achieves more than 91%recognition accuracy,lower false alarm rate and higher robustness compared with the multi-attribute decision-making(MADM)based on fuzzy numbers.

Missing Value Imputation for Radar-Derived Time-Series Tracks of Aerial Targets Based on Improved Self-Attention-Based Network

The frequent missing values in radar-derived time-series tracks of aerial targets(RTT-AT)lead to significant challenges in subsequent data-driven tasks.However,the majority of imputation research focuses on random missing(RM)that differs significantly from common missing patterns of RTT-AT.The method for solving the RM may experience performance degradation or failure when applied to RTT-AT imputation.Conventional autoregressive deep learning methods are prone to error accumulation and long-term dependency loss.In this paper,a non-autoregressive imputation model that addresses the issue of missing value imputation for two common missing patterns in RTT-AT is proposed.Our model consists of two probabilistic sparse diagonal masking self-attention(PSDMSA)units and a weight fusion unit.It learns missing values by combining the representations outputted by the two units,aiming to minimize the difference between the missing values and their actual values.The PSDMSA units effectively capture temporal dependencies and attribute correlations between time steps,improving imputation quality.The weight fusion unit automatically updates the weights of the output representations from the two units to obtain a more accurate final representation.The experimental results indicate that,despite varying missing rates in the two missing patterns,our model consistently outperforms other methods in imputation performance and exhibits a low frequency of deviations in estimates for specific missing entries.Compared to the state-of-the-art autoregressive deep learning imputation model Bidirectional Recurrent Imputation for Time Series(BRITS),our proposed model reduces mean absolute error(MAE)by 31%~50%.Additionally,the model attains a training speed that is 4 to 8 times faster when compared to both BRITS and a standard Transformer model when trained on the same dataset.Finally,the findings from the ablation experiments demonstrate that the PSDMSA,the weight fusion unit,cascade network design,and imputation loss enhance imputation performance and confirm the efficacy of our design.

Self-Attention Spatio-Temporal Deep Collaborative Network for Robust FDIA Detection in Smart Grids

False data injection attack(FDIA)can affect the state estimation of the power grid by tampering with the measured value of the power grid data,and then destroying the stable operation of the smart grid.Existing work usually trains a detection model by fusing the data-driven features from diverse power data streams.Data-driven features,however,cannot effectively capture the differences between noisy data and attack samples.As a result,slight noise disturbances in the power grid may cause a large number of false detections for FDIA attacks.To address this problem,this paper designs a deep collaborative self-attention network to achieve robust FDIA detection,in which the spatio-temporal features of cascaded FDIA attacks are fully integrated.Firstly,a high-order Chebyshev polynomials-based graph convolution module is designed to effectively aggregate the spatio information between grid nodes,and the spatial self-attention mechanism is involved to dynamically assign attention weights to each node,which guides the network to pay more attention to the node information that is conducive to FDIA detection.Furthermore,the bi-directional Long Short-Term Memory(LSTM)network is introduced to conduct time series modeling and long-term dependence analysis for power grid data and utilizes the temporal selfattention mechanism to describe the time correlation of data and assign different weights to different time steps.Our designed deep collaborative network can effectively mine subtle perturbations from spatiotemporal feature information,efficiently distinguish power grid noise from FDIA attacks,and adapt to diverse attack intensities.Extensive experiments demonstrate that our method can obtain an efficient detection performance over actual load data from New York Independent System Operator(NYISO)in IEEE 14,IEEE 39,and IEEE 118 bus systems,and outperforms state-of-the-art FDIA detection schemes in terms of detection accuracy and robustness.

Prediction and scheduling of multi-energy microgrid based on BiGRU self-attention mechanism and LQPSO

To predict renewable energy sources such as solar power in microgrids more accurately,a hybrid power prediction method is presented in this paper.First,the self-attention mechanism is introduced based on a bidirectional gated recurrent neural network(BiGRU)to explore the time-series characteristics of solar power output and consider the influence of different time nodes on the prediction results.Subsequently,an improved quantum particle swarm optimization(QPSO)algorithm is proposed to optimize the hyperparameters of the combined prediction model.The final proposed LQPSO-BiGRU-self-attention hybrid model can predict solar power more effectively.In addition,considering the coordinated utilization of various energy sources such as electricity,hydrogen,and renewable energy,a multi-objective optimization model that considers both economic and environmental costs was constructed.A two-stage adaptive multi-objective quantum particle swarm optimization algorithm aided by a Lévy flight,named MO-LQPSO,was proposed for the comprehensive optimal scheduling of a multi-energy microgrid system.This algorithm effectively balances the global and local search capabilities and enhances the solution of complex nonlinear problems.The effectiveness and superiority of the proposed scheme are verified through comparative simulations.

Stroke Electroencephalogram Data Synthesizing through Progressive Efficient Self-Attention Generative Adversarial Network

Early and timely diagnosis of stroke is critical for effective treatment,and the electroencephalogram(EEG)offers a low-cost,non-invasive solution.However,the shortage of high-quality patient EEG data often hampers the accuracy of diagnostic classification methods based on deep learning.To address this issue,our study designed a deep data amplification model named Progressive Conditional Generative Adversarial Network with Efficient Approximating Self Attention(PCGAN-EASA),which incrementally improves the quality of generated EEG features.This network can yield full-scale,fine-grained EEG features from the low-scale,coarse ones.Specially,to overcome the limitations of traditional generative models that fail to generate features tailored to individual patient characteristics,we developed an encoder with an effective approximating self-attention mechanism.This encoder not only automatically extracts relevant features across different patients but also reduces the computational resource consumption.Furthermore,the adversarial loss and reconstruction loss functions were redesigned to better align with the training characteristics of the network and the spatial correlations among electrodes.Extensive experimental results demonstrate that PCGAN-EASA provides the highest generation quality and the lowest computational resource usage compared to several existing approaches.Additionally,it significantly improves the accuracy of subsequent stroke classification tasks.

Aerial target threat assessment based on gated recurrent unit and self-attention mechanism

Aerial threat assessment is a crucial link in modern air combat, whose result counts a great deal for commanders to make decisions. With the consideration that the existing threat assessment methods have difficulties in dealing with high dimensional time series target data, a threat assessment method based on self-attention mechanism and gated recurrent unit(SAGRU) is proposed. Firstly, a threat feature system including air combat situations and capability features is established. Moreover, a data augmentation process based on fractional Fourier transform(FRFT) is applied to extract more valuable information from time series situation features. Furthermore, aiming to capture key characteristics of battlefield evolution, a bidirectional GRU and SA mechanisms are designed for enhanced features.Subsequently, after the concatenation of the processed air combat situation and capability features, the target threat level will be predicted by fully connected neural layers and the softmax classifier. Finally, in order to validate this model, an air combat dataset generated by a combat simulation system is introduced for model training and testing. The comparison experiments show the proposed model has structural rationality and can perform threat assessment faster and more accurately than the other existing models based on deep learning.

通过交叉验证堆栈和VAD信息检测Windows代码注入

Windows 32/64位代码注入攻击是恶意软件常用的攻击技术,在内存取证领域,现存的代码注入攻击检测技术在验证完整性方面不能处理动态内容,并且在解析内存中数据结构方面无法兼容不同版本的Windows系统。因此提出了通过交叉验证进程堆栈和VAD信息定位注入代码方法,将基于遍历栈帧得到的函数返回地址、模块名等信息结合进程VAD结构来检测函数返回地址、匹配文件名以定位注入代码,并且研发了基于Volatility取证框架的Windows代码注入攻击检测插件codefind。测试结果表明,即使在VAD节点被恶意软件修改,方法仍能够有效定位Windows 32/64位注入代码攻击。

CFSA-Net:Efficient Large-Scale Point Cloud Semantic Segmentation Based on Cross-Fusion Self-Attention

Traditional models for semantic segmentation in point clouds primarily focus on smaller scales.However,in real-world applications,point clouds often exhibit larger scales,leading to heavy computational and memory requirements.The key to handling large-scale point clouds lies in leveraging random sampling,which offers higher computational efficiency and lower memory consumption compared to other sampling methods.Nevertheless,the use of random sampling can potentially result in the loss of crucial points during the encoding stage.To address these issues,this paper proposes cross-fusion self-attention network(CFSA-Net),a lightweight and efficient network architecture specifically designed for directly processing large-scale point clouds.At the core of this network is the incorporation of random sampling alongside a local feature extraction module based on cross-fusion self-attention(CFSA).This module effectively integrates long-range contextual dependencies between points by employing hierarchical position encoding(HPC).Furthermore,it enhances the interaction between each point’s coordinates and feature information through cross-fusion self-attention pooling,enabling the acquisition of more comprehensive geometric information.Finally,a residual optimization(RO)structure is introduced to extend the receptive field of individual points by stacking hierarchical position encoding and cross-fusion self-attention pooling,thereby reducing the impact of information loss caused by random sampling.Experimental results on the Stanford Large-Scale 3D Indoor Spaces(S3DIS),Semantic3D,and SemanticKITTI datasets demonstrate the superiority of this algorithm over advanced approaches such as RandLA-Net and KPConv.These findings underscore the excellent performance of CFSA-Net in large-scale 3D semantic segmentation.

Clothing Parsing Based on Multi-Scale Fusion and Improved Self-Attention Mechanism

Due to the lack of long-range association and spatial location information,fine details and accurate boundaries of complex clothing images cannot always be obtained by using the existing deep learning-based methods.This paper presents a convolutional structure with multi-scale fusion to optimize the step of clothing feature extraction and a self-attention module to capture long-range association information.The structure enables the self-attention mechanism to directly participate in the process of information exchange through the down-scaling projection operation of the multi-scale framework.In addition,the improved self-attention module introduces the extraction of 2-dimensional relative position information to make up for its lack of ability to extract spatial position features from clothing images.The experimental results based on the colorful fashion parsing dataset(CFPD)show that the proposed network structure achieves 53.68%mean intersection over union(mIoU)and has better performance on the clothing parsing task.

基于Windows Server Failover Cluster的Apache故障转移集群应用实践

在当前互联网时代,Web应用程序的高可用性和可靠性是确保业务持续运行的关键因素。针对Apache Web服务器,介绍了利用Windows Server 2019 Failover Cluster构建Apache故障转移集群的实践方案。通过配置两个节点的故障转移集群,当某个节点发生故障时,Apache服务将自动在另一个节点上重新启动,从而最大程度地减少系统中断的时间,提高Web应用的可用性。详细阐述了集群搭建的具体步骤、配置要点、集群验证方法以及运行时的监控与维护方法,为构建高可用Apache Web服务器提供一种实用的解决方案。

基于Windows/RTX的实时仿测软件设计

针对传统测试软件实时性有限、传统仿真接口软件通用性程度低的问题,为满足半实物仿真软件需兼顾单元测试和控制系统仿真验证的需求,设计了基于Windows/RTX的实时仿测软件。仿测软件采用模块化的设计原则,开发了GUI层人机交互界面和RTX层实时运行程序。为保证实时性,采用无锁循环缓冲区+双线程技术,解决了RTX环境下仿真步长为1 ms时串口数据收发的超时问题;提出一种超时检测算法监测仿真节点的实时状态。借助cJSON优化了测试用例配置文件,用户可以更灵活地编辑测试用例,利用RTW自动代码生成将弹体模型编译集成到RTX仿测软件工程。实验结果表明:该仿测软件有效兼顾了单测与仿真,提高了仿测软件的通用性和二次开发效率,降低了开发难度。

基于Self-Attention的方面级情感分析方法研究

针对传统模型在细粒度的方面级情感分析上的不足,如RNN会遇到长距离依赖的问题,且模型不能并行计算;CNN的输出通常包含池化层,特征向量经过池化层的运算后会丢失相对位置信息和一些重要特征,且CNN没有考虑到文本的上下文信息。本文提出了一种Light-Transformer-ALSC模型,基于Self-Attention机制,且运用了交互注意力的思想,对方面词和上下文使用不同的注意力模块提取特征,细粒度地对文本进行情感分析,在SemEval2014 Task 4数据集上的实验结果表明本文模型的效果优于大部分仅基于LSTM的模型。除基于BERT的模型外,在Laptop数据集上准确率提高了1.3%~5.3%、在Restaurant数据集上准确率提高了2.5%~5.5%;对比基于BERT的模型,在准确率接近的情况下模型参数量大大减少。

Intelligent Fault Diagnosis Method of Rolling Bearings Based on Transfer Residual Swin Transformer with Shifted Windows

Due to their robust learning and expression ability for complex features,the deep learning(DL)model plays a vital role in bearing fault diagnosis.However,since there are fewer labeled samples in fault diagnosis,the depth of DL models in fault diagnosis is generally shallower than that of DL models in other fields,which limits the diagnostic performance.To solve this problem,a novel transfer residual Swin Transformer(RST)is proposed for rolling bearings in this paper.RST has 24 residual self-attention layers,which use the hierarchical design and the shifted window-based residual self-attention.Combined with transfer learning techniques,the transfer RST model uses pre-trained parameters from ImageNet.A new end-to-end method for fault diagnosis based on deep transfer RST is proposed.Firstly,wavelet transform transforms the vibration signal into a wavelet time-frequency diagram.The signal’s time-frequency domain representation can be represented simultaneously.Secondly,the wavelet time-frequency diagram is the input of the RST model to obtain the fault type.Finally,our method is verified on public and self-built datasets.Experimental results show the superior performance of our method by comparing it with a shallow neural network.

Bimetallic In_(2)O_(3)/Bi_(2)O_(3) Catalysts Enable Highly Selective CO_(2) Electroreduction to Formate within Ultra-Broad Potential Windows

CO_(2)electrochemical reduction reaction(CO_(2)RR)to formate is a hopeful pathway for reducing CO_(2)and producing high-value chemicals,which needs highly selective catalysts with ultra-broad potential windows to meet the industrial demands.Herein,the nanorod-like bimetallic ln_(2)O_(3)/Bi_(2)O_(3)catalysts were successfully synthesized by pyrolysis of bimetallic InBi-MOF precursors.The abundant oxygen vacancies generated from the lattice mismatch of Bi_(2)O_(3)and ln_(2)O_(3)reduced the activation energy of CO_(2)to*CO_(2)·^(-)and improved the selectivity of*CO_(2)·^(-)to formate simultaneously.Meanwhile,the carbon skeleton derived from the pyrolysis of organic framework of InBi-MOF provided a conductive network to accelerate the electrons transmission.The catalyst exhibited an ultra-broad applied potential window of 1200 mV(from-0.4 to-1.6 V vs RHE),relativistic high Faradaic efficiency of formate(99.92%)and satisfactory stability after 30 h.The in situ FT-IR experiment and DFT calculation verified that the abundant oxygen vacancies on the surface of catalysts can easily absorb CO_(2)molecules,and oxygen vacancy path is dominant pathway.This work provides a convenient method to construct high-performance bimetallic catalysts for the industrial application of CO_(2)RR.

Active and passive modulation of solar light transmittance in a uniquely multifunctional dual-band single molecule for smart window applications

Functional materials may change color by heat and electricity separately or simultaneously in smart windows.These materials have not only demonstrated remarkable potential in the modulation of solar radiation but are also leading to the development of indoor environments that are more comfortable and conducive to improving individuals'quality of life.Unfortunately,dual-responsive materials have not received ample research attention due to economic and technological challenges.As a consequence,the broader utilization of smart windows faces hindrances.To address this new generational multistimulus responsive chromic materials,our group has adopted a developmental strategy to create a poly(NIPAM)n-HV as a switchable material by anchoring active viologen(HV)onto a phase-changing poly(NIPAM)n-based smart material for better utility and activity.These constructed smart windows facilitate individualistic reversible switching,from a highly transparent state to an opaque state(thermochromic)and a red state(electrochromic),as well as facilitate a simultaneous dual-stimuli response reversible switching from a clear transparent state to a fully opaque(thermochromic)and orange(electrochromic)states.Absolute privacy can be attained in smart windows designed for exclusive settings by achieving zero transmittance.Each unique chromic mode operates independently and modulates visible and near-infrared(NIR)light in a distinct manner.Hence,these smart windows with thermal and electric dual-stimuli responsiveness demonstrate remarkable heat regulation capabilities,rendering them highly attractive for applications in building facades,energy harvesting,privacy protection,and color display.

Learning Epipolar Line Window Attention for Stereo Image Super-Resolution Reconstruction

Transformer-based stereo image super-resolution reconstruction(Stereo SR)methods have significantly improved image quality.However,existing methods have deficiencies in paying attention to detailed features and do not consider the offset of pixels along the epipolar lines in complementary views when integrating stereo information.To address these challenges,this paper introduces a novel epipolar line window attention stereo image super-resolution network(EWASSR).For detail feature restoration,we design a feature extractor based on Transformer and convolutional neural network(CNN),which consists of(shifted)window-based self-attention((S)W-MSA)and feature distillation and enhancement blocks(FDEB).This combination effectively solves the problem of global image perception and local feature attention and captures more discriminative high-frequency features of the image.Furthermore,to address the problem of offset of complementary pixels in stereo images,we propose an epipolar line window attention(EWA)mechanism,which divides windows along the epipolar direction to promote efficient matching of shifted pixels,even in pixel smooth areas.More accurate pixel matching can be achieved using adjacent pixels in the window as a reference.Extensive experiments demonstrate that our EWASSR can reconstruct more realistic detailed features.Comparative quantitative results show that in the experimental results of our EWASSR on the Middlebury and Flickr1024 data sets for 2×SR,compared with the recent network,the Peak signal-to-noise ratio(PSNR)increased by 0.37 dB and 0.34 dB,respectively.

AI-Driven Prioritization and Filtering of Windows Artifacts for Enhanced Digital Forensics

Digital forensics aims to uncover evidence of cybercrimes within compromised systems.These cybercrimes are often perpetrated through the deployment of malware,which inevitably leaves discernible traces within the compromised systems.Forensic analysts are tasked with extracting and subsequently analyzing data,termed as artifacts,from these systems to gather evidence.Therefore,forensic analysts must sift through extensive datasets to isolate pertinent evidence.However,manually identifying suspicious traces among numerous artifacts is time-consuming and labor-intensive.Previous studies addressed such inefficiencies by integrating artificial intelligence(AI)technologies into digital forensics.Despite the efforts in previous studies,artifacts were analyzed without considering the nature of the data within them and failed to prove their efficiency through specific evaluations.In this study,we propose a system to prioritize suspicious artifacts from compromised systems infected with malware to facilitate efficient digital forensics.Our system introduces a double-checking method that recognizes the nature of data within target artifacts and employs algorithms ideal for anomaly detection.The key ideas of this method are:(1)prioritize suspicious artifacts and filter remaining artifacts using autoencoder and(2)further prioritize suspicious artifacts and filter remaining artifacts using logarithmic entropy.Our evaluation demonstrates that our system can identify malicious artifacts with high accuracy and that its double-checking method is more efficient than alternative approaches.Our system can significantly reduce the time required for forensic analysis and serve as a reference for future studies.