Hierarchical multihead self-attention for time-series-based fault diagnosis

Fault diagnosis is important for maintaining the safety and effectiveness of chemical process.Considering the multivariate,nonlinear,and dynamic characteristic of chemical process,many time-series-based data-driven fault diagnosis methods have been developed in recent years.However,the existing methods have the problem of long-term dependency and are difficult to train due to the sequential way of training.To overcome these problems,a novel fault diagnosis method based on time-series and the hierarchical multihead self-attention(HMSAN)is proposed for chemical process.First,a sliding window strategy is adopted to construct the normalized time-series dataset.Second,the HMSAN is developed to extract the time-relevant features from the time-series process data.It improves the basic self-attention model in both width and depth.With the multihead structure,the HMSAN can pay attention to different aspects of the complicated chemical process and obtain the global dynamic features.However,the multiple heads in parallel lead to redundant information,which cannot improve the diagnosis performance.With the hierarchical structure,the redundant information is reduced and the deep local time-related features are further extracted.Besides,a novel many-to-one training strategy is introduced for HMSAN to simplify the training procedure and capture the long-term dependency.Finally,the effectiveness of the proposed method is demonstrated by two chemical cases.The experimental results show that the proposed method achieves a great performance on time-series industrial data and outperforms the state-of-the-art approaches.

SMSTracker:A Self-Calibration Multi-Head Self-Attention Transformer for Visual Object Tracking

Visual object tracking plays a crucial role in computer vision.In recent years,researchers have proposed various methods to achieve high-performance object tracking.Among these,methods based on Transformers have become a research hotspot due to their ability to globally model and contextualize information.However,current Transformer-based object tracking methods still face challenges such as low tracking accuracy and the presence of redundant feature information.In this paper,we introduce self-calibration multi-head self-attention Transformer(SMSTracker)as a solution to these challenges.It employs a hybrid tensor decomposition self-organizing multihead self-attention transformermechanism,which not only compresses and accelerates Transformer operations but also significantly reduces redundant data,thereby enhancing the accuracy and efficiency of tracking.Additionally,we introduce a self-calibration attention fusion block to resolve common issues of attention ambiguities and inconsistencies found in traditional trackingmethods,ensuring the stability and reliability of tracking performance across various scenarios.By integrating a hybrid tensor decomposition approach with a self-organizingmulti-head self-attentive transformer mechanism,SMSTracker enhances the efficiency and accuracy of the tracking process.Experimental results show that SMSTracker achieves competitive performance in visual object tracking,promising more robust and efficient tracking systems,demonstrating its potential to providemore robust and efficient tracking solutions in real-world applications.

A Self-Attention Based Dynamic Resource Management for Satellite-Terrestrial Networks

The satellite-terrestrial networks possess the ability to transcend geographical constraints inherent in traditional communication networks,enabling global coverage and offering users ubiquitous computing power support,which is an important development direction of future communications.In this paper,we take into account a multi-scenario network model under the coverage of low earth orbit(LEO)satellite,which can provide computing resources to users in faraway areas to improve task processing efficiency.However,LEO satellites experience limitations in computing and communication resources and the channels are time-varying and complex,which makes the extraction of state information a daunting task.Therefore,we explore the dynamic resource management issue pertaining to joint computing,communication resource allocation and power control for multi-access edge computing(MEC).In order to tackle this formidable issue,we undertake the task of transforming the issue into a Markov decision process(MDP)problem and propose the self-attention based dynamic resource management(SABDRM)algorithm,which effectively extracts state information features to enhance the training process.Simulation results show that the proposed algorithm is capable of effectively reducing the long-term average delay and energy consumption of the tasks.

Missing Value Imputation for Radar-Derived Time-Series Tracks of Aerial Targets Based on Improved Self-Attention-Based Network

The frequent missing values in radar-derived time-series tracks of aerial targets(RTT-AT)lead to significant challenges in subsequent data-driven tasks.However,the majority of imputation research focuses on random missing(RM)that differs significantly from common missing patterns of RTT-AT.The method for solving the RM may experience performance degradation or failure when applied to RTT-AT imputation.Conventional autoregressive deep learning methods are prone to error accumulation and long-term dependency loss.In this paper,a non-autoregressive imputation model that addresses the issue of missing value imputation for two common missing patterns in RTT-AT is proposed.Our model consists of two probabilistic sparse diagonal masking self-attention(PSDMSA)units and a weight fusion unit.It learns missing values by combining the representations outputted by the two units,aiming to minimize the difference between the missing values and their actual values.The PSDMSA units effectively capture temporal dependencies and attribute correlations between time steps,improving imputation quality.The weight fusion unit automatically updates the weights of the output representations from the two units to obtain a more accurate final representation.The experimental results indicate that,despite varying missing rates in the two missing patterns,our model consistently outperforms other methods in imputation performance and exhibits a low frequency of deviations in estimates for specific missing entries.Compared to the state-of-the-art autoregressive deep learning imputation model Bidirectional Recurrent Imputation for Time Series(BRITS),our proposed model reduces mean absolute error(MAE)by 31%~50%.Additionally,the model attains a training speed that is 4 to 8 times faster when compared to both BRITS and a standard Transformer model when trained on the same dataset.Finally,the findings from the ablation experiments demonstrate that the PSDMSA,the weight fusion unit,cascade network design,and imputation loss enhance imputation performance and confirm the efficacy of our design.

Prediction and scheduling of multi-energy microgrid based on BiGRU self-attention mechanism and LQPSO

To predict renewable energy sources such as solar power in microgrids more accurately,a hybrid power prediction method is presented in this paper.First,the self-attention mechanism is introduced based on a bidirectional gated recurrent neural network(BiGRU)to explore the time-series characteristics of solar power output and consider the influence of different time nodes on the prediction results.Subsequently,an improved quantum particle swarm optimization(QPSO)algorithm is proposed to optimize the hyperparameters of the combined prediction model.The final proposed LQPSO-BiGRU-self-attention hybrid model can predict solar power more effectively.In addition,considering the coordinated utilization of various energy sources such as electricity,hydrogen,and renewable energy,a multi-objective optimization model that considers both economic and environmental costs was constructed.A two-stage adaptive multi-objective quantum particle swarm optimization algorithm aided by a Lévy flight,named MO-LQPSO,was proposed for the comprehensive optimal scheduling of a multi-energy microgrid system.This algorithm effectively balances the global and local search capabilities and enhances the solution of complex nonlinear problems.The effectiveness and superiority of the proposed scheme are verified through comparative simulations.

A Pooling Method Developed for Use in Convolutional Neural Networks

In convolutional neural networks,pooling methods are used to reduce both the size of the data and the number of parameters after the convolution of the models.These methods reduce the computational amount of convolutional neural networks,making the neural network more efficient.Maximum pooling,average pooling,and minimum pooling methods are generally used in convolutional neural networks.However,these pooling methods are not suitable for all datasets used in neural network applications.In this study,a new pooling approach to the literature is proposed to increase the efficiency and success rates of convolutional neural networks.This method,which we call MAM(Maximum Average Minimum)pooling,is more interactive than other traditional maximum pooling,average pooling,and minimum pooling methods and reduces data loss by calculating the more appropriate pixel value.The proposed MAM pooling method increases the performance of the neural network by calculating the optimal value during the training of convolutional neural networks.To determine the success accuracy of the proposed MAM pooling method and compare it with other traditional pooling methods,training was carried out on the LeNet-5 model using CIFAR-10,CIFAR-100,and MNIST datasets.According to the results obtained,the proposed MAM pooling method performed better than the maximum pooling,average pooling,and minimum pooling methods in all pool sizes on three different datasets.

Effects of pooling,specialization,and discretionary task completion on queueing performance

Pooling,unpooling/specialization,and discretionary task completion are typical operational strategies in queueing systems that arise in healthcare,call centers,and online sales.These strategies may have advantages and disadvantages in different operational environments.This paper uses the M/M/1 and M/M/2 queues to study the impact of pooling,specialization,and discretionary task completion on the average queue length.Closed-form solutions for the average M/M/2 queue length are derived.Computational examples illustrate how the average queue length changes with the strength of pooling,specialization,and discretionary task completion.Finally,several conjectures are made in the paper.

Aerial target threat assessment based on gated recurrent unit and self-attention mechanism

Aerial threat assessment is a crucial link in modern air combat, whose result counts a great deal for commanders to make decisions. With the consideration that the existing threat assessment methods have difficulties in dealing with high dimensional time series target data, a threat assessment method based on self-attention mechanism and gated recurrent unit(SAGRU) is proposed. Firstly, a threat feature system including air combat situations and capability features is established. Moreover, a data augmentation process based on fractional Fourier transform(FRFT) is applied to extract more valuable information from time series situation features. Furthermore, aiming to capture key characteristics of battlefield evolution, a bidirectional GRU and SA mechanisms are designed for enhanced features.Subsequently, after the concatenation of the processed air combat situation and capability features, the target threat level will be predicted by fully connected neural layers and the softmax classifier. Finally, in order to validate this model, an air combat dataset generated by a combat simulation system is introduced for model training and testing. The comparison experiments show the proposed model has structural rationality and can perform threat assessment faster and more accurately than the other existing models based on deep learning.

Bayesian partial pooling to reduce uncertainty in overcoring rock stress estimation

The state of in situ stress is a crucial parameter in subsurface engineering,especially for critical projects like nuclear waste repository.As one of the two ISRM suggested methods,the overcoring(OC)method is widely used to estimate the full stress tensors in rocks by independent regression analysis of the data from each OC test.However,such customary independent analysis of individual OC tests,known as no pooling,is liable to yield unreliable test-specific stress estimates due to various uncertainty sources involved in the OC method.To address this problem,a practical and no-cost solution is considered by incorporating into OC data analysis additional information implied within adjacent OC tests,which are usually available in OC measurement campaigns.Hence,this paper presents a Bayesian partial pooling(hierarchical)model for combined analysis of adjacent OC tests.We performed five case studies using OC test data made at a nuclear waste repository research site of Sweden.The results demonstrate that partial pooling of adjacent OC tests indeed allows borrowing of information across adjacent tests,and yields improved stress tensor estimates with reduced uncertainties simultaneously for all individual tests than they are independently analysed as no pooling,particularly for those unreliable no pooling stress estimates.A further model comparison shows that the partial pooling model also gives better predictive performance,and thus confirms that the information borrowed across adjacent OC tests is relevant and effective.

U-Net Based Dual-Pooling Segmentation of Bone Metastases in Thoracic SPECT Bone Scintigrams

In order to enhance the performance of the CNN-based segmentation models for bone metastases, this study proposes a segmentation method that integrates dual-pooling, DAC, and RMP modules. The network consists of distinct feature encoding and decoding stages, with dual-pooling modules employed in encoding stages to maintain the background information needed for bone scintigrams diagnosis. Both the DAC and RMP modules are utilized in the bottleneck layer to address the multi-scale problem of metastatic lesions. Experimental evaluations on 306 clinical SPECT data have demonstrated that the proposed method showcases a substantial improvement in both DSC and Recall scores by 3.28% and 6.55% compared the baseline. Exhaustive case studies illustrate the superiority of the methodology.

CFSA-Net:Efficient Large-Scale Point Cloud Semantic Segmentation Based on Cross-Fusion Self-Attention

Traditional models for semantic segmentation in point clouds primarily focus on smaller scales.However,in real-world applications,point clouds often exhibit larger scales,leading to heavy computational and memory requirements.The key to handling large-scale point clouds lies in leveraging random sampling,which offers higher computational efficiency and lower memory consumption compared to other sampling methods.Nevertheless,the use of random sampling can potentially result in the loss of crucial points during the encoding stage.To address these issues,this paper proposes cross-fusion self-attention network(CFSA-Net),a lightweight and efficient network architecture specifically designed for directly processing large-scale point clouds.At the core of this network is the incorporation of random sampling alongside a local feature extraction module based on cross-fusion self-attention(CFSA).This module effectively integrates long-range contextual dependencies between points by employing hierarchical position encoding(HPC).Furthermore,it enhances the interaction between each point’s coordinates and feature information through cross-fusion self-attention pooling,enabling the acquisition of more comprehensive geometric information.Finally,a residual optimization(RO)structure is introduced to extend the receptive field of individual points by stacking hierarchical position encoding and cross-fusion self-attention pooling,thereby reducing the impact of information loss caused by random sampling.Experimental results on the Stanford Large-Scale 3D Indoor Spaces(S3DIS),Semantic3D,and SemanticKITTI datasets demonstrate the superiority of this algorithm over advanced approaches such as RandLA-Net and KPConv.These findings underscore the excellent performance of CFSA-Net in large-scale 3D semantic segmentation.

Clothing Parsing Based on Multi-Scale Fusion and Improved Self-Attention Mechanism

Due to the lack of long-range association and spatial location information,fine details and accurate boundaries of complex clothing images cannot always be obtained by using the existing deep learning-based methods.This paper presents a convolutional structure with multi-scale fusion to optimize the step of clothing feature extraction and a self-attention module to capture long-range association information.The structure enables the self-attention mechanism to directly participate in the process of information exchange through the down-scaling projection operation of the multi-scale framework.In addition,the improved self-attention module introduces the extraction of 2-dimensional relative position information to make up for its lack of ability to extract spatial position features from clothing images.The experimental results based on the colorful fashion parsing dataset(CFPD)show that the proposed network structure achieves 53.68%mean intersection over union(mIoU)and has better performance on the clothing parsing task.

基于Self-Attention的方面级情感分析方法研究

针对传统模型在细粒度的方面级情感分析上的不足,如RNN会遇到长距离依赖的问题,且模型不能并行计算;CNN的输出通常包含池化层,特征向量经过池化层的运算后会丢失相对位置信息和一些重要特征,且CNN没有考虑到文本的上下文信息。本文提出了一种Light-Transformer-ALSC模型,基于Self-Attention机制,且运用了交互注意力的思想,对方面词和上下文使用不同的注意力模块提取特征,细粒度地对文本进行情感分析,在SemEval2014 Task 4数据集上的实验结果表明本文模型的效果优于大部分仅基于LSTM的模型。除基于BERT的模型外,在Laptop数据集上准确率提高了1.3%~5.3%、在Restaurant数据集上准确率提高了2.5%~5.5%;对比基于BERT的模型,在准确率接近的情况下模型参数量大大减少。

Expanding the liver donor pool worldwide with hepatitis C infected livers, is it the time?

Liver transplantation(LT)provides a life-saving option for cirrhotic patients with complications and hepatocellular carcinoma.Despite the increasing number of liver transplants performed each year,the number of LT candidates on the waitlist remains unchanged due to an imbalance between donor organ supply and the demand which increases the waitlist time and mortality.Living donor liver transplant had a great role in increasing the donor pool and shortened waitlist time for LT candidates.Nevertheless,further strategies can be implemented to increase the pool of potential donors in deceased donor LT,such as reducing the rate of organ discards.Utilizing hepatitis C virus(HCV)seropositive liver grafts is one of the expanded donor organ criteria.A yearly increase of hundreds of transplants is anticipated as a result of maximizing the utilization of HCV-positive organs for HCV-negative recipients.Direct-acting antiviral therapy's efficacy has revolutionized the treatment of HCV infection and the use of HCV-seropositive donors in transplantation.The American Society of Transplantation advises against performing transplants from HCV-infected liver donors(D+)into HCV-negative recipient(R-)unless under Institutional Review Board-approved study rules and with full informed consent of the knowledge gaps associated with such transplants.Proper selection of patients to be transplanted with HCV-infected grafts and confirming their access to direct-acting antivirals if needed is im-portant.National and international consensuses are needed to regulate this process to ensure the maximum benefit and the least adverse events.

辛空间的排列问题及具有容错能力的pooling设计的紧界

该文利用辛空间上的子空间构造了一类新的d^z析取矩阵,然后研究了如下排列问题:对于给定的整数m,r,s,v,d,q和辛空间F_q^(2v)中的一个(m,s)型子空间S,这里v+s≥m>r≥2s-1≥1,d≥2,q是一个素数的幂,作者从S中找到d个(m-1,s-1)型子空间H_1,…H_d,使包含在这些(m-1,s-1)型子空间中的(r,s-1)型子空间个数达到最大.然后利用这个排列的有关结论,给出了一类pooling设计的紧界.

利用伪辛子空间构造的一类pooling设计及其纠错能力的紧界分析

利用伪辛空间上的子空间构造了一类新的dz-析取矩阵,证明了它的析取性,讨论了反映dz-析取矩阵纠错能力z值的紧界.

基于DNA pooling技术的全基因组关联研究筛选主动脉夹层等位基因遗传位点

目的:筛选与主动脉夹层发病机制相关的遗传易感基因。方法:从主动脉夹层患者(150例)及对照组(250例)外周血白细胞提取基因组DNA,采用DNA Pooling为基础的Illumina Human660W-Quad芯片扫描,筛选与主动脉夹层发病相关的遗传易感基因。结果:(1)对照组女性数量明显多于病例组(P<0.01);年龄、吸烟、高血压、糖尿病人数无统计学差异(P>0.05)。(2)遗传变异位点SNPrs2298491(位于TBCEL基因),SNP rs6080720(位于BFSP1基因),SNP rs7653410(位于SNTN基因),SNP rs2345106(位于COLQ基因)和位于ABCA13基因上的SNPrs4024044可能与主动脉夹层的发病有关。结论:SNPsrs2298491,rs6080720,rs7653410,rs2345106和rs4024044可能是主动脉夹层发病机制相关遗传变异位点的一部分。

奇特征正交空间上可检错Pooling设计的构作

Pooling设计的数学模型是一个d-disjunct矩阵.利用奇特征正交空间中全迷向子空间构作了d-disjunct矩阵,并通过计算它的Hamming距离分析了它的检纠错能力,根据Kautz-Singleton定理对d的范围作了估算.

基于Risk Pooling的Supply Hub库存优势分析

采用Risk Pooling的原理分析了Supply Hub运作模式相对于VMI分散运作模式给供应链带来优势。Supply Hub运作模式与VMI分散运作模式在同时向下游供应产品时,Supply Hub运作模式能以更低的安全库存和更少的库存成本来提供与VMI分散运作模式相同的服务水平。而且,Hub的规模越大,Supply Hub运作模式的这些优势就越明显。

Pooling PCR技术用于发现病程早期HIV感染者的研究

目的研究探讨PoolingPCR技术诊断识别病程早期(急性感染期)艾滋病病毒感染者的能力,并评估该方法是否经济可行、是否可以在部分人群的艾滋病病毒感染状态的诊断中取代艾滋病抗体检测方法。方法收集4家医疗机构留存的4类人群的HIV抗体阴性血样,对血样进行PoolingPCR法检测,并计算人均检测成本。结果男男同性恋人群HIV核酸阳性率为554.27/万,性病门诊就诊者人群次之,为不低于119.13/万,体检人群没有发现HIV核酸阳性;医疗机构1(结核病防治机构)的其他就诊者人群核酸阳性率(≥85.54/万)要高于医疗机构2的其他就诊者人群(≥21.04/万)。4个医疗卫生机构所留存的4类人群的抗体检测阴性样本中有2类人群检出HIV病毒核酸阳性,共检出病程早期(HIV抗体阴性,但核酸阳性)样本3例,其中男男同性恋人群检出2例、性病门诊就诊者人群检出1例。结论PoolingPCR检测技术方法敏感且经济,适合于在男男同性恋、性病门诊就诊者及结核病病人等HIV高危人群/高度可疑病人中发现病程早期HIV病毒感染者。