Masked Autoencoders as Single Object Tracking Learners

Significant advancements have beenwitnessed in visual tracking applications leveragingViT in recent years,mainly due to the formidablemodeling capabilities of Vision Transformer(ViT).However,the strong performance of such trackers heavily relies on ViT models pretrained for long periods,limitingmore flexible model designs for tracking tasks.To address this issue,we propose an efficient unsupervised ViT pretraining method for the tracking task based on masked autoencoders,called TrackMAE.During pretraining,we employ two shared-parameter ViTs,serving as the appearance encoder and motion encoder,respectively.The appearance encoder encodes randomly masked image data,while the motion encoder encodes randomly masked pairs of video frames.Subsequently,an appearance decoder and a motion decoder separately reconstruct the original image data and video frame data at the pixel level.In this way,ViT learns to understand both the appearance of images and the motion between video frames simultaneously.Experimental results demonstrate that ViT-Base and ViT-Large models,pretrained with TrackMAE and combined with a simple tracking head,achieve state-of-the-art(SOTA)performance without additional design.Moreover,compared to the currently popular MAE pretraining methods,TrackMAE consumes only 1/5 of the training time,which will facilitate the customization of diverse models for tracking.For instance,we additionally customize a lightweight ViT-XS,which achieves SOTA efficient tracking performance.

A Trust Evaluation Mechanism Based on Autoencoder Clustering Algorithm for Edge Device Access of IoT

First,we propose a cross-domain authentication architecture based on trust evaluation mechanism,including registration,certificate issuance,and cross-domain authentication processes.A direct trust evaluation mechanism based on the time decay factor is proposed,taking into account the influence of historical interaction records.We weight the time attenuation factor to each historical interaction record for updating and got the new historical record data.We refer to the beta distribution to enhance the flexibility and adaptability of the direct trust assessment model to better capture time trends in the historical record.Then we propose an autoencoder-based trust clustering algorithm.We perform feature extraction based on autoencoders.Kullback leibler(KL)divergence is used to calculate the reconstruction error.When constructing a convolutional autoencoder,we introduce convolutional neural networks to improve training efficiency and introduce sparse constraints into the hidden layer of the autoencoder.The sparse penalty term in the loss function measures the difference through the KL divergence.Trust clustering is performed based on the density based spatial clustering of applications with noise(DBSCAN)clustering algorithm.During the clustering process,edge nodes have a variety of trustworthy attribute characteristics.We assign different attribute weights according to the relative importance of each attribute in the clustering process,and a larger weight means that the attribute occupies a greater weight in the calculation of distance.Finally,we introduced adaptive weights to calculate comprehensive trust evaluation.Simulation experiments prove that our trust evaluation mechanism has excellent reliability and accuracy.

Trusted Encrypted Traffic Intrusion Detection Method Based on Federated Learning and Autoencoder

With the rapid development of the Internet,network security and data privacy are increasingly valued.Although classical Network Intrusion Detection System(NIDS)based on Deep Learning(DL)models can provide good detection accuracy,but collecting samples for centralized training brings the huge risk of data privacy leakage.Furthermore,the training of supervised deep learning models requires a large number of labeled samples,which is usually cumbersome.The“black-box”problem also makes the DL models of NIDS untrustworthy.In this paper,we propose a trusted Federated Learning(FL)Traffic IDS method called FL-TIDS to address the above-mentioned problems.In FL-TIDS,we design an unsupervised intrusion detection model based on autoencoders that alleviates the reliance on marked samples.At the same time,we use FL for model training to protect data privacy.In addition,we design an improved SHAP interpretable method based on chi-square test to perform interpretable analysis of the trained model.We conducted several experiments to evaluate the proposed FL-TIDS.We first determine experimentally the structure and the number of neurons of the unsupervised AE model.Secondly,we evaluated the proposed method using the UNSW-NB15 and CICIDS2017 datasets.The exper-imental results show that the unsupervised AE model has better performance than the other 7 intrusion detection models in terms of precision,recall and f1-score.Then,federated learning is used to train the intrusion detection model.The experimental results indicate that the model is more accurate than the local learning model.Finally,we use an improved SHAP explainability method based on Chi-square test to analyze the explainability.The analysis results show that the identification characteristics of the model are consistent with the attack characteristics,and the model is reliable.

Abnormal State Detection in Lithium-ion Battery Using Dynamic Frequency Memory and Correlation Attention LSTM Autoencoder

This paper addresses the challenge of identifying abnormal states in Lithium-ion Battery(LiB)time series data.As the energy sector increasingly focuses on integrating distributed energy resources,Virtual Power Plants(VPP)have become a vital new framework for energy management.LiBs are key in this context,owing to their high-efficiency energy storage capabilities essential for VPP operations.However,LiBs are prone to various abnormal states like overcharging,over-discharging,and internal short circuits,which impede power transmission efficiency.Traditional methods for detecting such abnormalities in LiB are too broad and lack precision for the dynamic and irregular nature of LiB data.In response,we introduce an innovative method:a Long Short-Term Memory(LSTM)autoencoder based on Dynamic Frequency Memory and Correlation Attention(DFMCA-LSTM-AE).This unsupervised,end-to-end approach is specifically designed for dynamically monitoring abnormal states in LiB data.The method starts with a Dynamic Frequency Fourier Transform module,which dynamically captures the frequency characteristics of time series data across three scales,incorporating a memory mechanism to reduce overgeneralization of abnormal frequencies.This is followed by integrating LSTM into both the encoder and decoder,enabling the model to effectively encode and decode the temporal relationships in the time series.Empirical tests on a real-world LiB dataset demonstrate that DFMCA-LSTM-AE outperforms existing models,achieving an average Area Under the Curve(AUC)of 90.73%and an F1 score of 83.83%.These results mark significant improvements over existing models,ranging from 2.4%–45.3%for AUC and 1.6%–28.9%for F1 score,showcasing the model’s enhanced accuracy and reliability in detecting abnormal states in LiB data.

A strategy for out-of-roundness damage wheels identification in railway vehicles based on sparse autoencoders

Wayside monitoring is a promising cost-effective alternative to predict damage in the rolling stock. The main goal of this work is to present an unsupervised methodology to identify out-of-roundness(OOR) damage wheels, such as wheel flats and polygonal wheels. This automatic damage identification algorithm is based on the vertical acceleration evaluated on the rails using a virtual wayside monitoring system and involves the application of a two-step procedure. The first step aims to define a confidence boundary by using(healthy) measurements evaluated on the rail constituting a baseline. The second step of the procedure involves classifying damage of predefined scenarios with different levels of severities. The proposed procedure is based on a machine learning methodology and includes the following stages:(1) data collection,(2) damage-sensitive feature extraction from the acquired responses using a neural network model, i.e., the sparse autoencoder(SAE),(3) data fusion based on the Mahalanobis distance, and(4) unsupervised feature classification by implementing outlier and cluster analysis. This procedure considers baseline responses at different speeds and rail irregularities to train the SAE model. Then, the trained SAE is capable to reconstruct test responses(not trained) allowing to compute the accumulative difference between original and reconstructed signals. The results prove the efficiency of the proposed approach in identifying the two most common types of OOR in railway wheels.

Network Intrusion Detection Model Based on Ensemble of Denoising Adversarial Autoencoder

Network security problems bring many imperceptible threats to the integrity of data and the reliability of device services,so proposing a network intrusion detection model with high reliability is of great research significance for network security.Due to the strong generalization of invalid features during training process,it is more difficult for single autoencoder intrusion detection model to obtain effective results.A network intrusion detection model based on the Ensemble of Denoising Adversarial Autoencoder(EDAAE)was proposed,which had higher accuracy and reliability compared to the traditional anomaly detection model.Using the adversarial learning idea of Adversarial Autoencoder(AAE),the discriminator module was added to the original model,and the encoder part was used as the generator.The distribution of the hidden space of the data generated by the encoder matched with the distribution of the original data.The generalization of the model to the invalid features was also reduced to improve the detection accuracy.At the same time,the denoising autoencoder and integrated operation was introduced to prevent overfitting in the adversarial learning process.Experiments on the CICIDS2018 traffic dataset showed that the proposed intrusion detection model achieves an Accuracy of 95.23%,which out performs traditional self-encoders and other existing intrusion detection models methods in terms of overall performance.

Generalized autoencoder-based fault detection method for traction systems with performance degradation

Fault diagnosis of traction systems is important for the safety operation of high-speed trains.Long-term operation of the trains will degrade the performance of systems,which decreases the fault detection accuracy.To solve this problem,this paper proposes a fault detection method developed by a Generalized Autoencoder(GAE)for systems with performance degradation.The advantage of this method is that it can accurately detect faults when the traction system of high-speed trains is affected by performance degradation.Regardless of the probability distribution,it can handle any data,and the GAE has extremely high sensitivity in anomaly detection.Finally,the effectiveness of this method is verified through the Traction Drive Control System(TDCS)platform.At different performance degradation levels,our method’s experimental results are superior to traditional methods.

Anomalous node detection in attributed social networks using dual variational autoencoder with generative adversarial networks

Many types of real-world information systems, including social media and e-commerce platforms, can be modelled by means of attribute-rich, connected networks. The goal of anomaly detection in artificial intelligence is to identify illustrations that deviate significantly from the main distribution of data or that differ from known cases. Anomalous nodes in node-attributed networks can be identified with greater precision if both graph and node attributes are taken into account. Almost all of the studies in this area focus on supervised techniques for spotting outliers. While supervised algorithms for anomaly detection work well in theory, they cannot be applied to real-world applications owing to a lack of labelled data. Considering the possible data distribution, our model employs a dual variational autoencoder (VAE), while a generative adversarial network (GAN) assures that the model is robust to adversarial training. The dual VAEs are used in another capacity: as a fake-node generator. Adversarial training is used to ensure that our latent codes have a Gaussian or uniform distribution. To provide a fair presentation of the graph, the discriminator instructs the generator to generate latent variables with distributions that are more consistent with the actual distribution of the data. Once the model has been learned, the discriminator is used for anomaly detection via reconstruction loss which has been trained to distinguish between the normal and artificial distributions of data. First, using a dual VAE, our model simultaneously captures cross-modality interactions between topological structure and node characteristics and overcomes the problem of unlabeled anomalies, allowing us to better understand the network sparsity and nonlinearity. Second, the proposed model considers the regularization of the latent codes while solving the issue of unregularized embedding techniques that can quickly lead to unsatisfactory representation. Finally, we use the discriminator reconstruction loss for anomaly detection as the discriminator is well-trained to separate the normal and generated data distributions because reconstruction-based loss does not include the adversarial component. Experiments conducted on attributed networks demonstrate the effectiveness of the proposed model and show that it greatly surpasses the previous methods. The area under the curve scores of our proposed model for the BlogCatalog, Flickr, and Enron datasets are 0.83680, 0.82020, and 0.71180, respectively, proving the effectiveness of the proposed model. The result of the proposed model on the Enron dataset is slightly worse than other models;we attribute this to the dataset’s low dimensionality as the most probable explanation.

融合Autoencoder方法的电力系统网络安全风险评估技术

针对现有电力系统安全评估指标存在高复杂度与低准确度等缺点,基于Autoencoder方法提出了适用于电力系统的网络安全风险评估方法.通过分析系统运行机制的脆弱性,使用层次分析法与专家调查法,建立了网络安全的初步评估指标体系模型.通过引入Autoencoder方法对复杂的指标体系模型进行必要的约简与优化,形成电力系统的新型安全评估模型.仿真结果表明,与传统的安全评估模型相比,所提出模型具有更高的执行效率与评估准确度.

A Lightweight Deep Autoencoder Scheme for Cyberattack Detection in the Internet of Things

The Internet of things(IoT)is an emerging paradigm that integrates devices and services to collect real-time data from surroundings and process the information at a very high speed to make a decision.Despite several advantages,the resource-constrained and heterogeneous nature of IoT networks makes them a favorite target for cybercriminals.A single successful attempt of network intrusion can compromise the complete IoT network which can lead to unauthorized access to the valuable information of consumers and industries.To overcome the security challenges of IoT networks,this article proposes a lightweight deep autoencoder(DAE)based cyberattack detection framework.The proposed approach learns the normal and anomalous data patterns to identify the various types of network intrusions.The most significant feature of the proposed technique is its lower complexity which is attained by reducing the number of operations.To optimally train the proposed DAE,a range of hyperparameters was determined through extensive experiments that ensure higher attack detection accuracy.The efficacy of the suggested framework is evaluated via two standard and open-source datasets.The proposed DAE achieved the accuracies of 98.86%,and 98.26%for NSL-KDD,99.32%,and 98.79%for the UNSW-NB15 dataset in binary class and multi-class scenarios.The performance of the suggested attack detection framework is also compared with several state-of-the-art intrusion detection schemes.Experimental outcomes proved the promising performance of the proposed scheme for cyberattack detection in IoT networks.

Bi-LSTM-Based Deep Stacked Sequence-to-Sequence Autoencoder for Forecasting Solar Irradiation and Wind Speed

Wind and solar energy are two popular forms of renewable energy used in microgrids and facilitating the transition towards net-zero carbon emissions by 2050.However,they are exceedingly unpredictable since they rely highly on weather and atmospheric conditions.In microgrids,smart energy management systems,such as integrated demand response programs,are permanently established on a step-ahead basis,which means that accu-rate forecasting of wind speed and solar irradiance intervals is becoming increasingly crucial to the optimal operation and planning of microgrids.With this in mind,a novel“bidirectional long short-term memory network”(Bi-LSTM)-based,deep stacked,sequence-to-sequence autoencoder(S2SAE)forecasting model for predicting short-term solar irradiation and wind speed was developed and evaluated in MATLAB.To create a deep stacked S2SAE prediction model,a deep Bi-LSTM-based encoder and decoder are stacked on top of one another to reduce the dimension of the input sequence,extract its features,and then reconstruct it to produce the forecasts.Hyperparameters of the proposed deep stacked S2SAE forecasting model were optimized using the Bayesian optimization algorithm.Moreover,the forecasting performance of the proposed Bi-LSTM-based deep stacked S2SAE model was compared to three other deep,and shallow stacked S2SAEs,i.e.,the LSTM-based deep stacked S2SAE model,gated recurrent unit-based deep stacked S2SAE model,and Bi-LSTM-based shallow stacked S2SAE model.All these models were also optimized and modeled in MATLAB.The results simulated based on actual data confirmed that the proposed model outperformed the alternatives by achieving an accuracy of up to 99.7%,which evidenced the high reliability of the proposed forecasting.

Rock mass quality classification based on deep learning:A feasibility study for stacked autoencoders

Objective and accurate evaluation of rock mass quality classification is the prerequisite for reliable sta-bility assessment.To develop a tool that can deliver quick and accurate evaluation of rock mass quality,a deep learning approach is developed,which uses stacked autoencoders(SAEs)with several autoencoders and a softmax net layer.Ten rock parameters of rock mass rating(RMR)system are calibrated in this model.The model is trained using 75%of the total database for training sample data.The SAEs trained model achieves a nearly 100%prediction accuracy.For comparison,other different models are also trained with the same dataset,using artificial neural network(ANN)and radial basis function(RBF).The results show that the SAEs classify all test samples correctly while the rating accuracies of ANN and RBF are 97.5%and 98.7%,repectively,which are calculated from the confusion matrix.Moreover,this model is further employed to predict the slope risk level of an abandoned quarry.The proposed approach using SAEs,or deep learning in general,is more objective and more accurate and requires less human inter-vention.The findings presented here shall shed light for engineers/researchers interested in analyzing rock mass classification criteria or performing field investigation.

Unsupervised Anomaly Detection Approach Based on Adversarial Memory Autoencoders for Multivariate Time Series

The widespread usage of Cyber Physical Systems(CPSs)generates a vast volume of time series data,and precisely determining anomalies in the data is critical for practical production.Autoencoder is the mainstream method for time series anomaly detection,and the anomaly is judged by reconstruction error.However,due to the strong generalization ability of neural networks,some abnormal samples close to normal samples may be judged as normal,which fails to detect the abnormality.In addition,the dataset rarely provides sufficient anomaly labels.This research proposes an unsupervised anomaly detection approach based on adversarial memory autoencoders for multivariate time series to solve the above problem.Firstly,an encoder encodes the input data into low-dimensional space to acquire a feature vector.Then,a memory module is used to learn the feature vector’s prototype patterns and update the feature vectors.The updating process allows partial forgetting of information to prevent model overgeneralization.After that,two decoders reconstruct the input data.Finally,this research uses the Peak Over Threshold(POT)method to calculate the threshold to determine anomalous samples from normal samples.This research uses a two-stage adversarial training strategy during model training to enlarge the gap between the reconstruction error of normal and abnormal samples.The proposed method achieves significant anomaly detection results on synthetic and real datasets from power systems,water treatment plants,and computer clusters.The F1 score reached an average of 0.9196 on the five datasets,which is 0.0769 higher than the best baseline method.

Hyperspectral anomaly detection via memory‐augmented autoencoders

Recently,the autoencoder(AE)based method plays a critical role in the hyperspectral anomaly detection domain.However,due to the strong generalised capacity of AE,the abnormal samples are usually reconstructed well along with the normal background samples.Thus,in order to separate anomalies from the background by calculating reconstruction errors,it can be greatly beneficial to reduce the AE capability for abnormal sample reconstruction while maintaining the background reconstruction performance.A memory‐augmented autoencoder for hyperspectral anomaly detection(MAENet)is proposed to address this challenging problem.Specifically,the proposed MAENet mainly consists of an encoder,a memory module,and a decoder.First,the encoder transforms the original hyperspectral data into the low‐dimensional latent representation.Then,the latent representation is utilised to retrieve the most relevant matrix items in the memory matrix,and the retrieved matrix items will be used to replace the latent representation from the encoder.Finally,the decoder is used to reconstruct the input hyperspectral data using the retrieved memory items.With this strategy,the background can still be reconstructed well while the abnormal samples cannot.Experiments conducted on five real hyperspectral anomaly data sets demonstrate the superiority of the proposed method.

A novel intrusion detection model for the CAN bus packet of in-vehicle network based on attention mechanism and autoencoder

The attacks on in-vehicle Controller Area Network(CAN)bus messages severely disrupt normal communication between vehicles.Therefore,researches on intrusion detection models for CAN have positive business value for vehicle security,and the intrusion detection technology for CAN bus messages can effectively protect the invehicle network from unlawful attacks.Previous machine learning-based models are unable to effectively identify intrusive abnormal messages due to their inherent shortcomings.Hence,to address the shortcomings of the previous machine learning-based intrusion detection technique,we propose a novel method using Attention Mechanism and AutoEncoder for Intrusion Detection(AMAEID).The AMAEID model first converts the raw hexadecimal message data into binary format to obtain better input.Then the AMAEID model encodes and decodes the binary message data using a multi-layer denoising autoencoder model to obtain a hidden feature representation that can represent the potential features behind the message data at a deeper level.Finally,the AMAEID model uses the attention mechanism and the fully connected layer network to infer whether the message is an abnormal message or not.The experimental results with three evaluation metrics on a real in-vehicle CAN bus message dataset outperform some traditional machine learning algorithms,demonstrating the effectiveness of the AMAEID model.

Constraint-Guided Autoencoders to Enforce a Predefined Threshold on Anomaly Scores:An Application in Machine Condition Monitoring

Anomaly detection(AD)is an important task in a broad range of domains.A popular choice for AD are Deep Support Vector Data Description models.When learning such models,normal data is mapped close to and anomalous data is mapped far from a center,in some latent space,enabling the construction of a sphere to separate both types of data.Empirically,it was observed:(i)that the center and radius of such sphere largely depend on the training data and model initialization which leads to difficulties when selecting a threshold,and(ii)that the center and radius of this sphere strongly impact the model AD performance on unseen data.In this work,a more robust AD solution is proposed that(i)defines a sphere with a fixed radius and margin in some latent space and(ii)enforces the encoder,which maps the input to a latent space,to encode the normal data in a small sphere and the anomalous data outside a larger sphere,with the same center.Experimental results indicate that the proposed algorithm attains higher performance compared to alternatives,and that the difference in size of the two spheres has a minor impact on the performance.

Network Intrusion Detection in Internet of Blended Environment Using Ensemble of Heterogeneous Autoencoders(E-HAE)

Contemporary attackers,mainly motivated by financial gain,consistently devise sophisticated penetration techniques to access important information or data.The growing use of Internet of Things(IoT)technology in the contemporary convergence environment to connect to corporate networks and cloud-based applications only worsens this situation,as it facilitates multiple new attack vectors to emerge effortlessly.As such,existing intrusion detection systems suffer from performance degradation mainly because of insufficient considerations and poorly modeled detection systems.To address this problem,we designed a blended threat detection approach,considering the possible impact and dimensionality of new attack surfaces due to the aforementioned convergence.We collectively refer to the convergence of different technology sectors as the internet of blended environment.The proposed approach encompasses an ensemble of heterogeneous probabilistic autoencoders that leverage the corresponding advantages of a convolutional variational autoencoder and long short-term memory variational autoencoder.An extensive experimental analysis conducted on the TON_IoT dataset demonstrated 96.02%detection accuracy.Furthermore,performance of the proposed approach was compared with various single model(autoencoder)-based network intrusion detection approaches:autoencoder,variational autoencoder,convolutional variational autoencoder,and long short-term memory variational autoencoder.The proposed model outperformed all compared models,demonstrating F1-score improvements of 4.99%,2.25%,1.92%,and 3.69%,respectively.

A Convolutional Autoencoder Based Fault Detection Method for Metro Railway Turnout

Railway turnout is one of the critical equipment of Switch&Crossing(S&C)Systems in railway,related to the train’s safety and operation efficiency.With the advancement of intelligent sensors,data-driven fault detection technology for railway turnout has become an important research topic.However,little research in the literature has investigated the capability of data-driven fault detection technology for metro railway turnout.This paper presents a convolutional autoencoder-based fault detection method for the metro railway turnout considering human field inspection scenarios.First,the one-dimensional original time-series signal is converted into a twodimensional image by data pre-processing and 2D representation.Next,a binary classification model based on the convolutional autoencoder is developed to implement fault detection.The profile and structure information can be captured by processing data as images.The performance of our method is evaluated and tested on real-world operational current data in themetro stations.Experimental results show that the proposedmethod achieves better performance,especially in terms of error rate and specificity,and is robust in practical engineering applications.

A Novel Scheme for Separate Training of Deep Learning-Based CSI Feedback Autoencoders

In this paper, we introduce a novel scheme for the separate training of deep learning-based autoencoders used for Channel State Information (CSI) feedback. Our distinct training approach caters to multiple users and base stations, enabling independent and individualized local training. This ensures the more secure processing of data and algorithms, different from the commonly adopted joint training method. To maintain comparable performance with joint training, we present two distinct training methods: separate training decoder and separate training encoder. It’s noteworthy that conducting separate training for the encoder can pose additional challenges, due to its responsibility in acquiring a compressed representation of underlying data features. This complexity makes accommodating multiple pre-trained decoders for just one encoder a demanding task. To overcome this, we design an adaptation layer architecture that effectively minimizes performance losses. Moreover, the flexible training strategy empowers users and base stations to seamlessly incorporate distinct encoder and decoder structures into the system, significantly amplifying the system’s scalability. .

Missing Value Imputation Model Based on Adversarial Autoencoder Using Spatiotemporal Feature Extraction

Recently,the importance of data analysis has increased significantly due to the rapid data increase.In particular,vehicle communication data,considered a significant challenge in Intelligent Transportation Systems(ITS),has spatiotemporal characteristics and many missing values.High missing values in data lead to the decreased predictive performance of models.Existing missing value imputation models ignore the topology of transportation net-works due to the structural connection of road networks,although physical distances are close in spatiotemporal image data.Additionally,the learning process of missing value imputation models requires complete data,but there are limitations in securing complete vehicle communication data.This study proposes a missing value imputation model based on adversarial autoencoder using spatiotemporal feature extraction to address these issues.The proposed method replaces missing values by reflecting spatiotemporal characteristics of transportation data using temporal convolution and spatial convolution.Experimental results show that the proposed model has the lowest error rate of 5.92%,demonstrating excellent predictive accuracy.Through this,it is possible to solve the data sparsity problem and improve traffic safety by showing superior predictive performance.