Changepoint Detection with Outliers Based on RWPCA

Changepoint detection faces challenges when outlier data are present. This paper proposes a multivariate changepoint detection method which is based on the robust WPCA projection direction and the robust RFPOP method, RWPCA-RFPOP method. Our method is double robust which is suitable for detecting mean changepoints in multivariate normal data with high correlations between variables that include outliers. Simulation results demonstrate that our method provides strong guarantees on both the number and location of changepoints in the presence of outliers. Finally, our method is well applied in an ACGH dataset.

Determination of uncertainties of geomechanical parameters of metamorphic rocks using petrographic analyses

Geomechanical parameters of intact metamorphic rocks determined from laboratory testing remain highly uncertain because of the great intrinsic variability associated with the degrees of metamorphism.The aim of this paper is to develop a proper methodology to analyze the uncertainties of geomechanical characteristics by focusing on three domains,i.e.data treatment process,schistosity angle,and mineralogy.First,the variabilities of the geomechanical laboratory data of Westwood Mine(Quebec,Canada)were examined statistically by applying different data treatment techniques,through which the most suitable outlier methods were selected for each parameter using multiple decision-making criteria and engineering judgment.Results indicated that some methods exhibited better performance in identifying the possible outliers,although several others were unsuccessful because of their limitation in large sample size.The well-known boxplot method might not be the best outlier method for most geomechanical parameters because its calculated confidence range was not acceptable according to engineering judgment.However,several approaches,including adjusted boxplot,2MADe,and 2SD,worked very well in the detection of true outliers.Also,the statistical tests indicate that the best-fitting probability distribution function for geomechanical intact parameters might not be the normal distribution,unlike what is assumed in most geomechanical studies.Moreover,the negative effects of schistosity angle on the uniaxial compressive strength(UCS)variabilities were reduced by excluding the samples within a specific angle range where the UCS data present the highest variation.Finally,a petrographic analysis was conducted to assess the associated uncertainties such that a logical link was found between the dispersion and the variabilities of hard and soft minerals.

Probabilistic modeling of multifunction radars with autoregressive kernel mixture network

The task of modeling and analyzing intercepted multifunction radars(MFRs)pulse trains is vital for cognitive electronic reconnaissance.Existing methodologies predominantly rely on prior information or heavily constrained models,posing challenges for non-cooperative applications.This paper introduces a novel approach to model MFRs using a Bayesian network,where the conditional probability density function is approximated by an autoregressive kernel mixture network(ARKMN).Utilizing the estimated probability density function,a dynamic programming algorithm is proposed for denoising and detecting change points in the intercepted MFRs pulse trains.Simulation results affirm the proposed method's efficacy in modeling MFRs,outperforming the state-of-the-art in pulse train denoising and change point detection.

Wavelet Based Detection of Outliers in Volatility Time Series Models

We introduce a new wavelet based procedure for detecting outliers in financial discrete time series.The procedure focuses on the analysis of residuals obtained from a model fit,and applied to the Generalized Autoregressive Conditional Heteroskedasticity(GARCH)like model,but not limited to these models.We apply the Maximal-Overlap Discrete Wavelet Transform(MODWT)to the residuals and compare their wavelet coefficients against quantile thresholds to detect outliers.Our methodology has several advantages over existing methods that make use of the standard Discrete Wavelet Transform(DWT).The series sample size does not need to be a power of 2 and the transform can explore any wavelet filter and be run up to the desired level.Simulated wavelet quantiles from a Normal and Student t-distribution are used as threshold for the maximum of the absolute value of wavelet coefficients.The performance of the procedure is illustrated and applied to two real series:the closed price of the Saudi Stock market and the S&P 500 index respectively.The efficiency of the proposed method is demonstrated and can be considered as a distinct important addition to the existing methods.

Identifying Extreme Rainfall Events Using Functional Outliers Detection Methods

Outlier detection techniques play a vital role in exploring unusual data of extreme events that have a critical effect considerably in the modeling and forecasting of functional data. The functional methods have an effective way of identifying outliers graphically, which might not be visible through the original data plot in classical analysis. This study’s main objective is to detect the extreme rainfall events using functional outliers detection methods depending on the depth and density functions. In order to identify the unusual events of rainfall variation over long time intervals, this work conducts based on the average monthly rainfall of the Taiz region from 1998 to 2019. Data were extracted from the Tropical Rainfall Measuring Mission and the analysis has been processed by R software. The approaches applied in this study involve rainbow plots, functional highest density region box-plot as well as functional bag-plot. According to the current results, the functional density box-plot method has proven effective in detecting outlier compared to the functional depth bag-plot method. In conclusion, the results of the current study showed that the rainfall over the Taiz region during the last two decades was influenced by the extreme events of years 1999, 2004, 2005, and 2009.

An Integrated Multilayered Framework for IoT Security Intrusion Decisions

Security breaches can seriously harm the Internet of Things(IoT)and Industrial IoT(IIoT)environments.The damage can exceedfinancial and material losses to threaten human lives.Overcoming these security risks is challenging given IoT ubiquity,complexity,and restricted resources.Security intrusion man-agement is a cornerstone in fortifying the defensive security process.This paper presents an integrated multilayered framework facilitating the orchestration of the security intrusion management process and developing security decision support systems.The proposed framework incorporates four layers with four dedicated processing phases.This paper focuses mainly on the analytical layer.We present the architecture and models for predictive intrusion analytics for reactive and proactive defense strategies.We differentiate between the device and network levels to master the complexity of IoT infrastructure.Benefiting from the singu-larity of IIoT devices traffic,we approach the reactive security intrusion predic-tion through outlier detection models mean.We thoroughly experiment with ten outlier detection models on the IIoT wustl realistic dataset.The obtained results show the adequacy of the approach with an area under the curve(AUC)results surpassing 98%for several models with a good level of precision and time effi-ciency.Furthermore,we investigate the use of survival analysis semi-parametric predictive models to forecast the security intrusion before its occurrence for the proactive security strategy.The experiments show encouraging results with a con-cordance index(c-Index)reaching 89%and an integrated brier score(IBS)of 0.02.By integrating outlier intrusion detection and survival forecasting,the fra-mework provides a valuable means to monitor the security intrusions in IoT.

CLOF Based Outlier Detection Algorithm of Temperature Data for Ethylene Cracking Furnace

The flue temperature is one of the important indicators to characterize the combustion state of an ethylene cracker furnace,the outliers of temperature data can lead to the false alarm.Conventional outlier detection algorithms such as the Isolation Forest algorithm and 3-sigma principle cannot detect the outliers accurately.In order to improve the detection accuracy and reduce the computational complexity,an outlier detection algorithm for flue temperature data based on the CLOF(Clipping Local Outlier Factor,CLOF)algorithm is proposed.The algorithm preprocesses the normalized data using the cluster pruning algorithm,and realizes the high accuracy and high efficiency outlier detection in the outliers candidate set.Using the flue temperature data of an ethylene cracking furnace in a petrochemical plant,the main parameters of the CLOF algorithm are selected according to the experimental results,and the outlier detection effect of the Isolation Forest algorithm,the 3-sigma principle,the conventional LOF algorithm and the CLOF algorithm are compared and analyzed.The results show that the appropriate clipping coefficient in the CLOF algorithm can significantly improve the detection efficiency and detection accuracy.Compared with the outlier detection results of the Isolation Forest algorithm and 3-sigma principle,the accuracy of the CLOF detection results is increased,and the amount of data calculation is significantly reduced.

Copy Move Forgery Detection Using Novel Quadsort Moth Flame Light Gradient Boosting Machine

A severe problem in modern information systems is Digital media tampering along with fake information.Even though there is an enhancement in image development,image forgery,either by the photographer or via image manipulations,is also done in parallel.Numerous researches have been concentrated on how to identify such manipulated media or information manually along with automatically;thus conquering the complicated forgery methodologies with effortlessly obtainable technologically enhanced instruments.However,high complexity affects the developed methods.Presently,it is complicated to resolve the issue of the speed-accuracy trade-off.For tackling these challenges,this article put forward a quick and effective Copy-Move Forgery Detection(CMFD)system utilizing a novel Quad-sort Moth Flame(QMF)Light Gradient Boosting Machine(QMF-Light GBM).Utilizing Borel Transform(BT)-based Wiener Filter(BWF)and resizing,the input images are initially pre-processed by eliminating the noise in the proposed system.After that,by utilizing the Orientation Preserving Simple Linear Iterative Clustering(OPSLIC),the pre-processed images,partitioned into a number of grids,are segmented.Next,as of the segmented images,the significant features are extracted along with the feature’s distance is calculated and matched with the input images.Next,utilizing the Union Topological Measure of Pattern Diversity(UTMOPD)method,the false positive matches that took place throughout the matching process are eliminated.After that,utilizing the QMF-Light GBM visualization,the visualization of forged in conjunction with non-forged images is performed.The extensive experiments revealed that concerning detection accuracy,the proposed system could be extremely precise when contrasted to some top-notch approaches.

Evolutionary Algorithm Based Feature Subset Selection for Students Academic Performance Analysis

Educational Data Mining(EDM)is an emergent discipline that concen-trates on the design of self-learning and adaptive approaches.Higher education institutions have started to utilize analytical tools to improve students’grades and retention.Prediction of students’performance is a difficult process owing to the massive quantity of educational data.Therefore,Artificial Intelligence(AI)techniques can be used for educational data mining in a big data environ-ment.At the same time,in EDM,the feature selection process becomes necessary in creation of feature subsets.Since the feature selection performance affects the predictive performance of any model,it is important to elaborately investigate the outcome of students’performance model related to the feature selection techni-ques.With this motivation,this paper presents a new Metaheuristic Optimiza-tion-based Feature Subset Selection with an Optimal Deep Learning model(MOFSS-ODL)for predicting students’performance.In addition,the proposed model uses an isolation forest-based outlier detection approach to eliminate the existence of outliers.Besides,the Chaotic Monarch Butterfly Optimization Algo-rithm(CBOA)is used for the selection of highly related features with low com-plexity and high performance.Then,a sailfish optimizer with stacked sparse autoencoder(SFO-SSAE)approach is utilized for the classification of educational data.The MOFSS-ODL model is tested against a benchmark student’s perfor-mance data set from the UCI repository.A wide-ranging simulation analysis por-trayed the improved predictive performance of the MOFSS-ODL technique over recent approaches in terms of different measures.Compared to other methods,experimental results prove that the proposed(MOFSS-ODL)classification model does a great job of predicting students’academic progress,with an accuracy of 96.49%.

Perceptual Optimization for Point-Based Point Cloud Rendering

Point-based rendering is a common method widely used in point cloud rendering.It realizes rendering by turning the points into the base geometry.The critical step in point-based rendering is to set an appropriate rendering radius for the base geometry,usually calculated using the average Euclidean distance of the N nearest neighboring points to the rendered point.This method effectively reduces the appearance of empty spaces between points in rendering.However,it also causes the problem that the rendering radius of outlier points far away from the central region of the point cloud sequence could be large,which impacts the perceptual quality.To solve the above problem,we propose an algorithm for point-based point cloud rendering through outlier detection to optimize the perceptual quality of rendering.The algorithm determines whether the detected points are outliers using a combination of local and global geometric features.For the detected outliers,the minimum radius is used for rendering.We examine the performance of the proposed method in terms of both objective quality and perceptual quality.The experimental results show that the peak signal-to-noise ratio(PSNR)of the point cloud sequences is improved under all geometric quantization,and the PSNR improvement ratio is more evident in dense point clouds.Specifically,the PSNR of the point cloud sequences is improved by 3.6%on average compared with the original algorithm.The proposed method significantly improves the perceptual quality of the rendered point clouds and the results of ablation studies prove the feasibility and effectiveness of the proposed method.

Analysis of Salaries and Some Non-traditional Measures of Location

The paper deals with an analysis of how to use certain measures of location in analysis of salaries. One of the traditional measures of location, the mean should offer typical value of variable, representing all its values by the best way. Sometimes, the mean is located in the tail of the distribution and gives a very biased idea about the location of the distribution. In these cases, using different measures of location could be useful. Trimmed mean is described. The trimmed mean refers to a situation where a certain proportion of the largest and smallest observations are removed and the remaining observations are averaged. The construction of some measures of location is based on the analysis of outliers. Outliers are characterized. Then the possibilities of the detection of outliers are analyzed. Computing of one-step M-estimator and modified one-step M-estimator of location is described. A comparison of the trimmed means and M-estimators of location is presented. Finally, the paper focuses on the application of the trimmed mean and M-estimators of location in analysis of salaries. The analysis of salaries of employers of the big Slovak companies in second half of the year 2009 is realized. The data from the census are used in the analysis. The median, 20% trimmed mean and the characteristics, based on the one-step M-estimator of location and modified one step M-estimator, are calculated.

Density-based trajectory outlier detection algorithm

With the development of global position system（GPS）,wireless technology and location aware services,it is possible to collect a large quantity of trajectory data.In the field of data mining for moving objects,the problem of anomaly detection is a hot topic.Based on the development of anomalous trajectory detection of moving objects,this paper introduces the classical trajectory outlier detection（TRAOD） algorithm,and then proposes a density-based trajectory outlier detection（DBTOD） algorithm,which compensates the disadvantages of the TRAOD algorithm that it is unable to detect anomalous defects when the trajectory is local and dense.The results of employing the proposed algorithm to Elk1993 and Deer1995 datasets are also presented,which show the effectiveness of the algorithm.

Packet Cache-Forward Method Based on Improved Bayesian Outlier Detection for Mobile Handover in Satellite Networks

In this paper, we propose a Packet Cache-Forward(PCF) method based on improved Bayesian outlier detection to eliminate out-of-order packets caused by transmission path drastically degradation during handover events in the moving satellite networks, for improving the performance of TCP. The proposed method uses an access node satellite to cache all received packets in a short time when handover occurs and forward them out in order. To calculate the cache time accurately, this paper establishes the Bayesian based mixture model for detecting delay outliers of the entire handover scheme. In view of the outliers' misjudgment, an updated classification threshold and the sliding window has been suggested to correct category collections and model parameters for the purpose of quickly identifying exact compensation delay in the varied network load statuses. Simulation shows that, comparing to average processing delay detection method, the average accuracy rate was scaled up by about 4.0%, and there is about 5.5% cut in error rate in the meantime. It also behaves well even though testing with big dataset. Benefiting from the advantage of the proposed scheme in terms of performance, comparing to conventional independent handover and network controlled synchronizedhandover in simulated LEO satellite networks, the proposed independent handover with PCF eliminates packet out-of-order issue to get better improvement on congestion window. Eventually the average delay decreases more than 70% and TCP performance has improved more than 300%.

GA-iForest: An Efficient Isolated Forest Framework Based on Genetic Algorithm for Numerical Data Outlier Detection

With the development of data age,data quality has become one of the problems that people pay much attention to.As a field of data mining,outlier detection is related to the quality of data.The isolated forest algorithm is one of the more prominent numerical data outlier detection algorithms in recent years.In the process of constructing the isolation tree by the isolated forest algorithm,as the isolation tree is continuously generated,the difference of isolation trees will gradually decrease or even no difference,which will result in the waste of memory and reduced efficiency of outlier detection.And in the constructed isolation trees,some isolation trees cannot detect outlier.In this paper,an improved iForest-based method GA-iForest is proposed.This method optimizes the isolated forest by selecting some better isolation trees according to the detection accuracy and the difference of isolation trees,thereby reducing some duplicate,similar and poor detection isolation trees and improving the accuracy and stability of outlier detection.In the experiment,Ubuntu system and Spark platform are used to build the experiment environment.The outlier datasets provided by ODDS are used as test.According to indicators such as the accuracy,recall rate,ROC curves,AUC and execution time,the performance of the proposed method is evaluated.Experimental results show that the proposed method can not only improve the accuracy and stability of outlier detection,but also reduce the number of isolation trees by 20%-40%compared with the original iForest method.

Probabilistic outlier detection for sparse multivariate geotechnical site investigation data using Bayesian learning

Various uncertainties arising during acquisition process of geoscience data may result in anomalous data instances(i.e.,outliers)that do not conform with the expected pattern of regular data instances.With sparse multivariate data obtained from geotechnical site investigation,it is impossible to identify outliers with certainty due to the distortion of statistics of geotechnical parameters caused by outliers and their associated statistical uncertainty resulted from data sparsity.This paper develops a probabilistic outlier detection method for sparse multivariate data obtained from geotechnical site investigation.The proposed approach quantifies the outlying probability of each data instance based on Mahalanobis distance and determines outliers as those data instances with outlying probabilities greater than 0.5.It tackles the distortion issue of statistics estimated from the dataset with outliers by a re-sampling technique and accounts,rationally,for the statistical uncertainty by Bayesian machine learning.Moreover,the proposed approach also suggests an exclusive method to determine outlying components of each outlier.The proposed approach is illustrated and verified using simulated and real-life dataset.It showed that the proposed approach properly identifies outliers among sparse multivariate data and their corresponding outlying components in a probabilistic manner.It can significantly reduce the masking effect(i.e.,missing some actual outliers due to the distortion of statistics by the outliers and statistical uncertainty).It also found that outliers among sparse multivariate data instances affect significantly the construction of multivariate distribution of geotechnical parameters for uncertainty quantification.This emphasizes the necessity of data cleaning process(e.g.,outlier detection)for uncertainty quantification based on geoscience data.

Efficient and Effective 4D Trajectory Data Cleansing

As the rapid development of aviation industry and newly emerging crowd-sourcing projects such as Flightradar24 and FlightAware,large amount of air traffic data,particularly four-dimension(4D)trajectory data,have become available for the public.In order to guarantee the accuracy and reliability of results,data cleansing is the first step in analyzing 4D trajectory data,including error identification and mitigation.Data cleansing techniques for the 4D trajectory data are investigated.Back propagation(BP)neural network algorithm is applied to repair errors.Newton interpolation method is used to obtain even-spaced trajectory samples over a uniform distribution of each flight’s 4D trajectory data.Furthermore,a new method is proposed to compress data while maintaining the intrinsic characteristics of the trajectories.Density-based spatial clustering of applications with noise(DBSCAN)is applied to identify remaining outliers of sample points.Experiments are performed on a data set of one-day 4D trajectory data over Europe.The results show that the proposed method can achieve more efficient and effective results than the existing approaches.The work contributes to the first step of data preprocessing and lays foundation for further downstream 4D trajectory analysis.

Outlier Detection for Water Supply Data Based on Joint Auto-Encoder

With the development of science and technology,the status of the water environment has received more and more attention.In this paper,we propose a deep learning model,named a Joint Auto-Encoder network,to solve the problem of outlier detection in water supply data.The Joint Auto-Encoder network first expands the size of training data and extracts the useful features from the input data,and then reconstructs the input data effectively into an output.The outliers are detected based on the network’s reconstruction errors,with a larger reconstruction error indicating a higher rate to be an outlier.For water supply data,there are mainly two types of outliers:outliers with large values and those with values closed to zero.We set two separate thresholds,and,for the reconstruction errors to detect the two types of outliers respectively.The data samples with reconstruction errors exceeding the thresholds are voted to be outliers.The two thresholds can be calculated by the classification confusion matrix and the receiver operating characteristic(ROC)curve.We have also performed comparisons between the Joint Auto-Encoder and the vanilla Auto-Encoder in this paper on both the synthesis data set and the MNIST data set.As a result,our model has proved to outperform the vanilla Auto-Encoder and some other outlier detection approaches with the recall rate of 98.94 percent in water supply data.

Outlier Behavior Detection for Indoor Environment Based on t-SNE Clustering

In this study,we propose a low-cost system that can detect the space outlier utilization of residents in an indoor environment.We focus on the users’app usage to analyze unusual behavior,especially in indoor spaces.This is reflected in the behavioral analysis in that the frequency of using smartphones in personal spaces has recently increased.Our system facilitates autonomous data collection from mobile app logs and Google app servers and generates a high-dimensional dataset that can detect outlier behaviors.The density-based spatial clustering of applications with noise(DBSCAN)algorithm was applied for effective singular movement analysis.To analyze high-level mobile phone usage,the t-distributed stochastic neighbor embedding(t-SNE)algorithm was employed.These two clustering algorithms can effectively detect outlier behaviors in terms of movement and app usage in indoor spaces.The experimental results showed that our system enables effective spatial behavioral analysis at a low cost when applied to logs collected in actual living spaces.Moreover,large volumes of data required for outlier detection can be easily acquired.The system can automatically detect the unusual behavior of a user in an indoor space.In particular,this study aims to reflect the recent trend of the increasing use of smartphones in indoor spaces to the behavioral analysis.

On-line outlier and change point detection for time series

The detection of outliers and change points from time series has become research focus in the area of time series data mining since it can be used for fraud detection, rare event discovery, event/trend change detection, etc. In most previous works, outlier detection and change point detection have not been related explicitly and the change point detections did not consider the influence of outliers, in this work, a unified detection framework was presented to deal with both of them. The framework is based on ALARCON-AQUINO and BARRIA's change points detection method and adopts two-stage detection to divide the outliers and change points. The advantages of it lie in that: firstly, unified structure for change detection and outlier detection further reduces the computational complexity and make the detective procedure simple; Secondly, the detection strategy of outlier detection before change point detection avoids the influence of outliers to the change point detection, and thus improves the accuracy of the change point detection. The simulation experiments of the proposed method for both model data and actual application data have been made and gotten 100% detection accuracy. The comparisons between traditional detection method and the proposed method further demonstrate that the unified detection structure is more accurate when the time series are contaminated by outliers.

A Hybrid Deep Learning-Based Unsupervised Anomaly Detection in High Dimensional Data

Anomaly detection in high dimensional data is a critical research issue with serious implication in the real-world problems.Many issues in this field still unsolved,so several modern anomaly detection methods struggle to maintain adequate accuracy due to the highly descriptive nature of big data.Such a phenomenon is referred to as the“curse of dimensionality”that affects traditional techniques in terms of both accuracy and performance.Thus,this research proposed a hybrid model based on Deep Autoencoder Neural Network(DANN)with five layers to reduce the difference between the input and output.The proposed model was applied to a real-world gas turbine(GT)dataset that contains 87620 columns and 56 rows.During the experiment,two issues have been investigated and solved to enhance the results.The first is the dataset class imbalance,which solved using SMOTE technique.The second issue is the poor performance,which can be solved using one of the optimization algorithms.Several optimization algorithms have been investigated and tested,including stochastic gradient descent(SGD),RMSprop,Adam and Adamax.However,Adamax optimization algorithm showed the best results when employed to train theDANNmodel.The experimental results show that our proposed model can detect the anomalies by efficiently reducing the high dimensionality of dataset with accuracy of 99.40%,F1-score of 0.9649,Area Under the Curve(AUC)rate of 0.9649,and a minimal loss function during the hybrid model training.