Multivariate Time Series Anomaly Detection Based on Spatial-Temporal Network and Transformer in Industrial Internet of Things

In the Industrial Internet of Things(IIoT),sensors generate time series data to reflect the working state.When the systems are attacked,timely identification of outliers in time series is critical to ensure security.Although many anomaly detection methods have been proposed,the temporal correlation of the time series over the same sensor and the state(spatial)correlation between different sensors are rarely considered simultaneously in these methods.Owing to the superior capability of Transformer in learning time series features.This paper proposes a time series anomaly detection method based on a spatial-temporal network and an improved Transformer.Additionally,the methods based on graph neural networks typically include a graph structure learning module and an anomaly detection module,which are interdependent.However,in the initial phase of training,since neither of the modules has reached an optimal state,their performance may influence each other.This scenario makes the end-to-end training approach hard to effectively direct the learning trajectory of each module.This interdependence between the modules,coupled with the initial instability,may cause the model to find it hard to find the optimal solution during the training process,resulting in unsatisfactory results.We introduce an adaptive graph structure learning method to obtain the optimal model parameters and graph structure.Experiments on two publicly available datasets demonstrate that the proposed method attains higher anomaly detection results than other methods.

AFSTGCN:Prediction for multivariate time series using an adaptive fused spatial-temporal graph convolutional network

The prediction for Multivariate Time Series(MTS)explores the interrelationships among variables at historical moments,extracts their relevant characteristics,and is widely used in finance,weather,complex industries and other fields.Furthermore,it is important to construct a digital twin system.However,existing methods do not take full advantage of the potential properties of variables,which results in poor predicted accuracy.In this paper,we propose the Adaptive Fused Spatial-Temporal Graph Convolutional Network(AFSTGCN).First,to address the problem of the unknown spatial-temporal structure,we construct the Adaptive Fused Spatial-Temporal Graph(AFSTG)layer.Specifically,we fuse the spatial-temporal graph based on the interrelationship of spatial graphs.Simultaneously,we construct the adaptive adjacency matrix of the spatial-temporal graph using node embedding methods.Subsequently,to overcome the insufficient extraction of disordered correlation features,we construct the Adaptive Fused Spatial-Temporal Graph Convolutional(AFSTGC)module.The module forces the reordering of disordered temporal,spatial and spatial-temporal dependencies into rule-like data.AFSTGCN dynamically and synchronously acquires potential temporal,spatial and spatial-temporal correlations,thereby fully extracting rich hierarchical feature information to enhance the predicted accuracy.Experiments on different types of MTS datasets demonstrate that the model achieves state-of-the-art single-step and multi-step performance compared with eight other deep learning models.

Advancing Autoencoder Architectures for Enhanced Anomaly Detection in Multivariate Industrial Time Series

In the context of rapid digitization in industrial environments,how effective are advanced unsupervised learning models,particularly hybrid autoencoder models,at detecting anomalies in industrial control system(ICS)datasets?This study is crucial because it addresses the challenge of identifying rare and complex anomalous patterns in the vast amounts of time series data generated by Internet of Things(IoT)devices,which can significantly improve the reliability and safety of these systems.In this paper,we propose a hybrid autoencoder model,called ConvBiLSTMAE,which combines convolutional neural network(CNN)and bidirectional long short-term memory(BiLSTM)to more effectively train complex temporal data patterns in anomaly detection.On the hardware-in-the-loopbased extended industrial control system dataset,the ConvBiLSTM-AE model demonstrated remarkable anomaly detection performance,achieving F1 scores of 0.78 and 0.41 for the first and second datasets,respectively.The results suggest that hybrid autoencoder models are not only viable,but potentially superior alternatives for unsupervised anomaly detection in complex industrial systems,offering a promising approach to improving their reliability and safety.

Fine-Grained Multivariate Time Series Anomaly Detection in IoT

Sensors produce a large amount of multivariate time series data to record the states of Internet of Things(IoT)systems.Multivariate time series timestamp anomaly detection(TSAD)can identify timestamps of attacks and malfunctions.However,it is necessary to determine which sensor or indicator is abnormal to facilitate a more detailed diagnosis,a process referred to as fine-grained anomaly detection(FGAD).Although further FGAD can be extended based on TSAD methods,existing works do not provide a quantitative evaluation,and the performance is unknown.Therefore,to tackle the FGAD problem,this paper first verifies that the TSAD methods achieve low performance when applied to the FGAD task directly because of the excessive fusion of features and the ignoring of the relationship’s dynamic changes between indicators.Accordingly,this paper proposes a mul-tivariate time series fine-grained anomaly detection(MFGAD)framework.To avoid excessive fusion of features,MFGAD constructs two sub-models to independently identify the abnormal timestamp and abnormal indicator instead of a single model and then combines the two kinds of abnormal results to detect the fine-grained anomaly.Based on this framework,an algorithm based on Graph Attention Neural Network(GAT)and Attention Convolutional Long-Short Term Memory(A-ConvLSTM)is proposed,in which GAT learns temporal features of multiple indicators to detect abnormal timestamps and A-ConvLSTM captures the dynamic relationship between indicators to identify abnormal indicators.Extensive simulations on a real-world dataset demonstrate that the proposed algorithm can achieve a higher F1 score and hit rate than the extension of existing TSAD methods with the benefit of two independent sub-models for timestamp and indicator detection.

A Memory-Guided Anomaly Detection Model with Contrastive Learning for Multivariate Time Series

Some reconstruction-based anomaly detection models in multivariate time series have brought impressive performance advancements but suffer from weak generalization ability and a lack of anomaly identification.These limitations can result in the misjudgment of models,leading to a degradation in overall detection performance.This paper proposes a novel transformer-like anomaly detection model adopting a contrastive learning module and a memory block(CLME)to overcome the above limitations.The contrastive learning module tailored for time series data can learn the contextual relationships to generate temporal fine-grained representations.The memory block can record normal patterns of these representations through the utilization of attention-based addressing and reintegration mechanisms.These two modules together effectively alleviate the problem of generalization.Furthermore,this paper introduces a fusion anomaly detection strategy that comprehensively takes into account the residual and feature spaces.Such a strategy can enlarge the discrepancies between normal and abnormal data,which is more conducive to anomaly identification.The proposed CLME model not only efficiently enhances the generalization performance but also improves the ability of anomaly detection.To validate the efficacy of the proposed approach,extensive experiments are conducted on well-established benchmark datasets,including SWaT,PSM,WADI,and MSL.The results demonstrate outstanding performance,with F1 scores of 90.58%,94.83%,91.58%,and 91.75%,respectively.These findings affirm the superiority of the CLME model over existing stateof-the-art anomaly detection methodologies in terms of its ability to detect anomalies within complex datasets accurately.

Dynamic Ensemble Multivariate Time Series Forecasting Model for PM2.5

In forecasting real time environmental factors,large data is needed to analyse the pattern behind the data values.Air pollution is a major threat towards developing countries and it is proliferating every year.Many methods in time ser-ies prediction and deep learning models to estimate the severity of air pollution.Each independent variable contributing towards pollution is necessary to analyse the trend behind the air pollution in that particular locality.This approach selects multivariate time series and coalesce a real time updatable autoregressive model to forecast Particulate matter(PM)PM2.5.To perform experimental analysis the data from the Central Pollution Control Board(CPCB)is used.Prediction is car-ried out for Chennai with seven locations and estimated PM’s using the weighted ensemble method.Proposed method for air pollution prediction unveiled effective and moored performance in long term prediction.Dynamic budge with high weighted k-models are used simultaneously and devising an ensemble helps to achieve stable forecasting.Computational time of ensemble decreases with paral-lel processing in each sub model.Weighted ensemble model shows high perfor-mance in long term prediction when compared to the traditional time series models like Vector Auto-Regression(VAR),Autoregressive Integrated with Mov-ing Average(ARIMA),Autoregressive Moving Average with Extended terms(ARMEX).Evaluation metrics like Root Mean Square Error(RMSE),Mean Absolute Error(MAE)and the time to achieve the time series are compared.

基于MTS-BiGRU-DMHSA的工业负荷预测方法

工业用电占我国全社会用电量比重大,通过工业负荷预测了解负荷趋势和用电量信息,有助于电网安全稳定运行,为电力部门发电规划提供依据,且有助于工业用户优化生产工艺和降低成本。为了兼顾工业负荷波动的不确定性以及工业用户用电行为的规律性特征,提出一种基于多时间尺度(MTS)特征的工业负荷预测方法MTS-BiGRU-DMHSA,利用MTS特征融合挖掘工业负荷的周期趋势特征和局部波动特征,提升工业负荷表征的可解释性。此外,双层多头自注意力(DMHSA)机制利用注意力权重聚焦重要特征,在挖掘输入特征关联性的同时捕捉时序关联性,强化重要特征变量与关键时间步的信息表达。在中国某工业企业五面受总柜实采数据上完成实验验证,采用2种评价指标对所提方法及5种基于神经网络的预测方法进行对比分析。实验结果表明,所提方法相较于对比方法平均误差降低逾20%,其中4.67%得益于对MTS特征的运用。通过对比各方法计算效率证实了所提方法的综合性能优势,可视化实验结果与对比分析也验证了其在工业负荷预测任务上的可行性。

A prediction comparison between univariate and multivariate chaotic time series

The methods to determine time delays and embedding dimensions in the phase space delay reconstruction of multivariate chaotic time series are proposed. Three nonlinear prediction methods of multivariate chaotic time series including local mean prediction, local linear prediction and BP neural networks prediction are considered. The simulation results obtained by the Lorenz system show that no matter what nonlinear prediction method is used, the prediction error of multivariate chaotic time series is much smaller than the prediction error of univariate time series, even if half of the data of univariate time series are used in multivariate time series. The results also verify that methods to determine the time delays and the embedding dimensions are correct from the view of minimizing the prediction error.

Generating Adversarial Samples on Multivariate Time Series using Variational Autoencoders

Classification models for multivariate time series have drawn the interest of many researchers to the field with the objective of developing accurate and efficient models.However,limited research has been conducted on generating adversarial samples for multivariate time series classification models.Adversarial samples could become a security concern in systems with complex sets of sensors.This study proposes extending the existing gradient adversarial transformation network(GATN)in combination with adversarial autoencoders to attack multivariate time series classification models.The proposed model attacks classification models by utilizing a distilled model to imitate the output of the multivariate time series classification model.In addition,the adversarial generator function is replaced with a variational autoencoder to enhance the adversarial samples.The developed methodology is tested on two multivariate time series classification models:1-nearest neighbor dynamic time warping(1-NN DTW)and a fully convolutional network(FCN).This study utilizes 30 multivariate time series benchmarks provided by the University of East Anglia(UEA)and University of California Riverside(UCR).The use of adversarial autoencoders shows an increase in the fraction of successful adversaries generated on multivariate time series.To the best of our knowledge,this is the first study to explore adversarial attacks on multivariate time series.Additionally,we recommend future research utilizing the generated latent space from the variational autoencoders.

Multivariate time series prediction based on AR_CLSTM

Time series is a kind of data widely used in various fields such as electricity forecasting,exchange rate forecasting,and solar power generation forecasting,and therefore time series prediction is of great significance.Recently,the encoder-decoder model combined with long short-term memory(LSTM)is widely used for multivariate time series prediction.However,the encoder can only encode information into fixed-length vectors,hence the performance of the model decreases rapidly as the length of the input sequence or output sequence increases.To solve this problem,we propose a combination model named AR_CLSTM based on the encoder_decoder structure and linear autoregression.The model uses a time step-based attention mechanism to enable the decoder to adaptively select past hidden states and extract useful information,and then uses convolution structure to learn the internal relationship between different dimensions of multivariate time series.In addition,AR_CLSTM combines the traditional linear autoregressive method to learn the linear relationship of the time series,so as to further reduce the error of time series prediction in the encoder_decoder structure and improve the multivariate time series Predictive effect.Experiments show that the AR_CLSTM model performs well in different time series predictions,and its root mean square error,mean square error,and average absolute error all decrease significantly.

Generic reconstruction technology based on RST for multivariate time series of complex process industries

In order to effectively analyse the multivariate time series data of complex process,a generic reconstruction technology based on reduction theory of rough sets was proposed,Firstly,the phase space of multivariate time series was originally reconstructed by a classical reconstruction technology.Then,the original decision-table of rough set theory was set up according to the embedding dimensions and time-delays of the original reconstruction phase space,and the rough set reduction was used to delete the redundant dimensions and irrelevant variables and to reconstruct the generic phase space,Finally,the input vectors for the prediction of multivariate time series were extracted according to generic reconstruction results to identify the parameters of prediction model.Verification results show that the developed reconstruction method leads to better generalization ability for the prediction model and it is feasible and worthwhile for application.

Production performance forecasting method based on multivariate time series and vector autoregressive machine learning model for waterflooding reservoirs

A forecasting method of oil well production based on multivariate time series(MTS)and vector autoregressive(VAR)machine learning model for waterflooding reservoir is proposed,and an example application is carried out.This method first uses MTS analysis to optimize injection and production data on the basis of well pattern analysis.The oil production of different production wells and water injection of injection wells in the well group are regarded as mutually related time series.Then a VAR model is established to mine the linear relationship from MTS data and forecast the oil well production by model fitting.The analysis of history production data of waterflooding reservoirs shows that,compared with history matching results of numerical reservoir simulation,the production forecasting results from the machine learning model are more accurate,and uncertainty analysis can improve the safety of forecasting results.Furthermore,impulse response analysis can evaluate the oil production contribution of the injection well,which can provide theoretical guidance for adjustment of waterflooding development plan.

Volatility in High-Frequency Intensive Care Mortality Time Series: Application of Univariate and Multivariate GARCH Models

Mortality time series display time-varying volatility. The utility of statistical estimators from the financial time-series paradigm, which account for this characteristic, has not been addressed for high-frequency mortality series. Using daily mean-mortality series of an exemplar intensive care unit (ICU) from the Australian and New Zealand Intensive Care Society adult patient database, joint estimation of a mean and conditional variance (volatility) model for a stationary series was undertaken via univariate autoregressive moving average (ARMA, lags (p, q)), GARCH (Generalised Autoregressive Conditional Heteroscedasticity, lags (p, q)). The temporal dynamics of the conditional variance and correlations of multiple provider series, from rural/ regional, metropolitan, tertiary and private ICUs, were estimated utilising multivariate GARCH models. For the stationary first differenced series, an asymmetric power GARCH model (lags (1, 1)) with t distribution (degrees-of- freedom, 11.6) and ARMA (7,0) for the mean-model, was the best-fitting. The four multivariate component series demonstrated varying trend mortality decline and persistent autocorrelation. Within each MGARCH series no model specification dominated. The conditional correlations were surprisingly low (<0.1) between tertiary series and substantial (0.4 - 0.6) between rural-regional and private series. The conditional-variances of both the univariate and multivariate series demonstrated a slow rate of time decline from periods of early volatility and volatility spikes.

Research on Pattern Matching Method of Multivariate Hydrological Time Series

The existing pattern matching methods of multivariate time series can hardly measure the similarity of multivariate hydrological time series accurately and efficiently.Considering the characteristics of multivariate hydrological time series,the continuity and global features of variables,we proposed a pattern matching method,PP-DTW,which is based on dynamic time warping.In this method,the multivariate time series is firstly segmented,and the average of each segment is used as the feature.Then,PCA is operated on the feature sequence.Finally,the weighted DTW distance is used as the measure of similarity in sequences.Carrying out experiments on the hydrological data of Chu River,we conclude that the pattern matching method can effectively describe the overall characteristics of the multivariate time series,which has a good matching effect on the multivariate hydrological time series.

Computational Method for Extracting and Modeling Periodicities in Time Series

Periodicity is common in natural processes, however, extraction tools are typically difficult and cumbersome to use. Here we report a computational method developed in MATLAB through a function called Periods with the aim to find the main harmonic components of time series data. This function is designed to obtain the period, amplitude and lag phase of the main harmonic components in a time series (Periods and lag phase components can be related to climate, social or economic events). It is based on methods of periodic regression with cyclic descent and includes statistical significance testing. The proposed method is very easy to use. Furthermore, it does not require full understanding of time series theory, nor require many inputs from the user. However, it is sufficiently flexible to undertake more complex tasks such as forecasting. Additionally, based on previous knowledge, specific periods can be included or excluded easily. The output results are organized into two groups. One contains the parameters of the adjusted model and their F statistics. The other consists of the harmonic parameters that best fit the original series according to their importance and the summarized statistics of the comparisons between successive models in the cyclic descent process. Periods is tested with both, simulated and actual sunspot and Multivariate ENSO Index data to show its performance and accuracy.

基于TimesNet的大类资产指数长期预测

大类资产指数预测常用于衡量投资组合的收益表现,若能对其进行准确的长期预测,则可据此构建投资组合以获取更高的收益。与传统的机器学习和常见的神经网络相比,TimesNet在处理多变量长期预测任务方面具有更好的效果。文章基于10个大类资产指数的历史数据,采用TimesNet模型进行长期预测,并在实证分析中与线性回归、随机森林、支持向量回归和LSTM四种预测模型进行对比,验证了该模型的准确性。

时间序列异常检测方法研究综述

时间序列是按时间顺序排列的一组数据点或观测值,在金融学、气象学和股票市场分析等领域中被广泛应用。时间序列数据出现异常可能意味着出现潜在问题、异常事件或系统故障。为了便于未来在时间序列异常检测方法设计方面开展深入研究,本文首先介绍时间序列异常检测的相关概念;其次,展开分析国内外单变量和多变量时间序列异常检测方法;之后,介绍一些时间序列异常检测通用数据集并比较常见检测方法在这些数据集上的性能;最后,探讨未来时间序列异常检测方法设计的重点研究方向,以期对相关理论和应用研究提供参考。

基于双向稀疏Transformer的多变量时序分类模型

针对多变量时序(Multivariate Time Series,MTS)分类中长序列数据难以捕捉时序特征的问题,提出一种基于双向稀疏Transformer的时序分类模型BST(Bidirectional Sparse Transformer),提高了MTS分类任务的准确度.BST模型使用Transformer框架,构建了一种基于活跃度得分的双向稀疏注意力机制.基于KL散度构建活跃度评价函数,并将评价函数的非对称问题转变为对称权重问题.据此,对原有查询矩阵、键值矩阵进行双向稀疏化,从而降低原Transformer模型中自注意力机制运算的时间复杂度.实验结果显示,BST模型在9个长序列数据集上取得最高平均排名,在临界差异图中领先第2名35.7%,对于具有强时序性的乙醇浓度数据集(Ethanol Concentration,EC),分类准确率提高30.9%.

结合重构和图预测的多元时序异常检测框架

高维时序异常检测一直是智能系统安全领域的重要挑战,主流解决方案通常使用基于数据降维的重构方法和基于时序建模的预测方法,但这些方法没有结合特征间相互影响和特征内时间关联进行学习,且大多使用点估计方法进行预测或重构,从而影响了异常检测的准确性。结合预测和重构的优点,考虑序列的整体分布,提出了一种新颖的端到端异常检测框架。设计改进的变分自动编码器重构模块,以学习原始时序数据中的特征内时间关联,同时得到编码后的低维表示。设计估计高斯分布的图神经网络预测模块,结合重构模块的低维表示和原始输入进行图结构学习,以捕捉特征间的结构依赖。模型采用异常评分模块联合重构和预测模块的损失,在考虑序列整体分布的基础上进行时空联合表征。为验证所提出模型的性能,在三个工业数据集上对模型进行了对比实验,与基线模型相比,所提出的模型在F1性能指标上表现良好。

用于多元时间序列预测的图神经网络模型

现有用于多元时序预测的图神经网络模型大多基于预定义图以静态的方式捕捉时序特征,缺少对于系统动态适应和对时序样本之间潜在动态关系的捕捉.提出用于多元时序预测的图神经网络模型(MTSGNN).该模型在一个图学习模块中,采用数据驱动的方式学习时间序列数据的静态图和动态演化图,以捕捉时序样本之间的复杂关系.通过图交互模块实现静态图和动态图之间的信息交互,并使用卷积运算提取交互信息中的依赖关系.利用多层感知机对多元时序进行预测.实验结果表明,所提模型在6个真实的多元时间序列数据集上的预测效果显著优于当前最先进的方法,并且具有参数量较小、运算速度较快的优点.