Changepoint Detection with Outliers Based on RWPCA

Changepoint detection faces challenges when outlier data are present. This paper proposes a multivariate changepoint detection method which is based on the robust WPCA projection direction and the robust RFPOP method, RWPCA-RFPOP method. Our method is double robust which is suitable for detecting mean changepoints in multivariate normal data with high correlations between variables that include outliers. Simulation results demonstrate that our method provides strong guarantees on both the number and location of changepoints in the presence of outliers. Finally, our method is well applied in an ACGH dataset.

A Study of Detection of Outliers for Working and Non-Working Days Air Quality in Kolkata, India: A Case Study

A variety of factors affect air quality, making it a difficult issue. The level of clean air in a certain area is referred to as air quality. It is challenging for conventional approaches to correctly discover aberrant values or outliers due to the significant fluctuation of this sort of data, which is influenced by Climate change and the environment. With accelerating industrial expansion and rising population density in Kolkata City, air pollution is continuously rising. This study involves two phases, in the first phase imputation of missing values and second detection of outliers using Statistical Process Control (SPC), and Functional Data Analysis (FDA), studies to achieve the efficacy of the outlier identification methodology proposed with working days and Nonworking days of the variables NO2, SO2, and O3, which were used for a year in a row in Kolkata, India. The results show how the functional data approach outshines traditional outlier detection methods. The outcomes show that functional data analysis vibrates more than the other two approaches after imputation, and the suggested outlier detector is absolutely appropriate for the precise detection of outliers in highly variable data.

CLOF Based Outlier Detection Algorithm of Temperature Data for Ethylene Cracking Furnace

The flue temperature is one of the important indicators to characterize the combustion state of an ethylene cracker furnace,the outliers of temperature data can lead to the false alarm.Conventional outlier detection algorithms such as the Isolation Forest algorithm and 3-sigma principle cannot detect the outliers accurately.In order to improve the detection accuracy and reduce the computational complexity,an outlier detection algorithm for flue temperature data based on the CLOF(Clipping Local Outlier Factor,CLOF)algorithm is proposed.The algorithm preprocesses the normalized data using the cluster pruning algorithm,and realizes the high accuracy and high efficiency outlier detection in the outliers candidate set.Using the flue temperature data of an ethylene cracking furnace in a petrochemical plant,the main parameters of the CLOF algorithm are selected according to the experimental results,and the outlier detection effect of the Isolation Forest algorithm,the 3-sigma principle,the conventional LOF algorithm and the CLOF algorithm are compared and analyzed.The results show that the appropriate clipping coefficient in the CLOF algorithm can significantly improve the detection efficiency and detection accuracy.Compared with the outlier detection results of the Isolation Forest algorithm and 3-sigma principle,the accuracy of the CLOF detection results is increased,and the amount of data calculation is significantly reduced.

A Novel Outlier Detection with Feature Selection Enabled Streaming Data Classification

Due to the advancements in information technologies,massive quantity of data is being produced by social media,smartphones,and sensor devices.The investigation of data stream by the use of machine learning(ML)approaches to address regression,prediction,and classification problems have received consid-erable interest.At the same time,the detection of anomalies or outliers and feature selection(FS)processes becomes important.This study develops an outlier detec-tion with feature selection technique for streaming data classification,named ODFST-SDC technique.Initially,streaming data is pre-processed in two ways namely categorical encoding and null value removal.In addition,Local Correla-tion Integral(LOCI)is used which is significant in the detection and removal of outliers.Besides,red deer algorithm(RDA)based FS approach is employed to derive an optimal subset of features.Finally,kernel extreme learning machine(KELM)classifier is used for streaming data classification.The design of LOCI based outlier detection and RDA based FS shows the novelty of the work.In order to assess the classification outcomes of the ODFST-SDC technique,a series of simulations were performed using three benchmark datasets.The experimental results reported the promising outcomes of the ODFST-SDC technique over the recent approaches.

Outliers rejection in similar image matching

Background Image matching is crucial in numerous computer vision tasks such as 3D reconstruction and simultaneous visual localization and mapping.The accuracy of the matching significantly impacted subsequent studies.Because of their local similarity,when image pairs contain comparable patterns but feature pairs are positioned differently,incorrect recognition can occur as global motion consistency is disregarded.Methods This study proposes an image-matching filtering algorithm based on global motion consistency.It can be used as a subsequent matching filter for the initial matching results generated by other matching algorithms based on the principle of motion smoothness.A particular matching algorithm can first be used to perform the initial matching;then,the rotation and movement information of the global feature vectors are combined to effectively identify outlier matches.The principle is that if the matching result is accurate,the feature vectors formed by any matched point should have similar rotation angles and moving distances.Thus,global motion direction and global motion distance consistencies were used to reject outliers caused by similar patterns in different locations.Results Four datasets were used to test the effectiveness of the proposed method.Three datasets with similar patterns in different locations were used to test the results for similar images that could easily be incorrectly matched by other algorithms,and one commonly used dataset was used to test the results for the general image-matching problem.The experimental results suggest that the proposed method is more accurate than other state-of-the-art algorithms in identifying mismatches in the initial matching set.Conclusions The proposed outlier rejection matching method can significantly improve the matching accuracy for similar images with locally similar feature pairs in different locations and can provide more accurate matching results for subsequent computer vision tasks.

Outlier Detection of Air Quality for Two Indian Urban Cities Using Functional Data Analysis

Human living would be impossible without air quality. Consistent advancements in practically every aspect of contemporary human life have harmed air quality. Everyday industrial, transportation, and home activities turn up dangerous contaminants in our surroundings. This study investigated two years’ worth of air quality and outlier detection data from two Indian cities. Studies on air pollution have used numerous types of methodologies, with various gases being seen as a vector whose components include gas concentration values for each observation per-formed. We use curves to represent the monthly average of daily gas emissions in our technique. The approach, which is based on functional depth, was used to find outliers in the city of Delhi and Kolkata’s gas emissions, and the outcomes were compared to those from the traditional method. In the evaluation and comparison of these models’ performances, the functional approach model studied well.

基于期望核密度离群因子的离群点检测算法

针对基于密度的离群点检测方法在不同分布的数据集上检测精度低的问题,提出了一种基于期望核密度离群因子的离群点检测算法。首先,引入k近邻和反向k近邻扩展邻域空间(ENS)代替传统的k邻域范围,更加全面地考虑数据对象的邻域信息;其次,在传统核密度估计(KDE)方法的基础上引入多元高斯函数,在扩展邻域空间内估计数据对象的密度,同时借鉴自适应核带宽的思想,更好地适应不同数据集的数据分布;然后,给出期望距离的概念,进一步区分局部离群点和位于低密度区域的正常点;最后,定义了期望核密度离群因子刻画数据对象离群程度。在人工数据集和真实数据集上对所提算法进行实验验证,并与部分传统算法进行对比,验证了所提算法的有效性。

基于贡献度和数据有效性检验的共识机制

将区块链技术引入到分布式数据维护系统,旨在解决基于传统中心化数据库的分布式系统存在的数据维护不透明、数据易被篡改、历史记录不可追溯等问题,提出一种基于贡献度和数据有效性检验的共识机制.该算法提出一种贡献度优先的随机可验证领导者选举机制,保证记账权分配的随机性及可验证性.进一步引入密度峰值算法对交易数据有效性进行校验,对打包区块的正确性达成共识.最后将所提出的共识机制应用于梅花鹿分布式养殖场场景,结果验证了密度峰值算法在交易数据有效性检测任务中的准确性和高效性.出块时延分析和安全性分析表明,所提出的共识机制能够满足数据有效性验证的实时性需求,能耗较小,具有很强的灾备能力.

基于5G移动通信技术在医疗设备管理中的研究

目的 为解决现有医疗设备通信技术滞后、设备管理能力欠佳等问题,提出一种基于5G移动通信技术在医疗设备管理中的方法,以提高医疗设备之间的数据信息通信能力。方法 基于5G通信技术,融合安全传输层协议,对传输数据进行加密;结合WebRTC技术,实现远程视频通话和会诊;使用离群检测算法对数据进行处理,提高通信数据的信息检索能力;加入网络波动检测模块,降低在远程访问中网络波动带来的影响。结果 通过记录某医疗机构某年1—10月的远程视频次数及网络波动次数和触发网络加速模块的次数可知,本系统将目标特征误差保持在2%以下,在系统运行过程中非常稳定且适用。结论 本系统大大减轻了系统主机的压力,具有较高的工作效率及较低的误差率和波动率,可达到远程医用和远程急救的要求。

基于映射距离比离群因子的离群点检测算法

针对基于邻近性的离群点检测方法需要花费大量时间过滤正常点,并且在检测全局离群点时难以检测出局部离群点的问题,提出一种基于映射距离比离群因子离群点检测(MDROF)算法。首先,为了减少正常点在检测过程中的时间消耗,给出了差异相似度的概念,通过定义差异相似度剪枝因子过滤掉数据集中的大部分正常点。其次,定义映射k距离,通过映射距离与可达距离的比值刻画数据对象的局部离群程度,通过可达密度刻画数据对象的全局离群程度。最后,结合数据对象相互近邻点的平均排位定义映射距离比离群因子来检测离群点。在人工数据集以及真实数据集上分别对该算法与其他经典的离群点检测算法在精确率、AUC值和离群点发现曲线上进行实验对比分析。实验结果证明MDROF算法在离群点检测的准确性和稳定性上明显优于对比算法。

基于自适应高斯渐进滤波的工程车GNSS/INS紧组合定位

研究了量测野值影响下的工程车GNSS/INS紧组合定位问题,提出了一种基于自适应高斯渐进滤波的车辆定位方法。首先,为降低量测野值对滤波器的破坏风险,利用假设检验方法对量测野值进行检测和剔除;其次,对于野值漏检测引起的定位性能下降的问题,设计了自适应的高斯渐进滤波方法来补偿量测的不确定性;特别地,利用线性化误差与系统估计误差的变化关系,对渐进量测更新方式进行了改进,从而实现对线性化误差的间接补偿。最后,通过工程车GNSS/INS紧组合定位实验进行结果分析,验证了所提方法的可靠性和优越性。

启发式k-means聚类算法的改进研究

启发式k-means聚类算法通过在k-means第一次迭代后查看附近的集群来预测每个数据点可能会被划分到的集群子集,有效地加快了算法的运行速度。但由于启发式算法存在随机选择初始聚类中心以及无法有效识别数据集中离群点的缺陷,导致聚类结果的误差平方和较大并且轮廓系数偏小。针对这一问题,提出了CHk-means算法,该算法引入仔细播种方法,克服了启发式k-means算法随机选择初始聚类中心带来的局部最优解问题;该算法引入局部异常因子LOF算法对离群点进行检测,降低了离群点数据对聚类结果的影响。在多个数据集上对3种算法进行对比试验,结果表明CHk-means算法可有效降低聚类结果的误差平方和,增强聚类的轮廓系数,使聚类质量得到明显改善。

高校高价值专利技术机会识别研究——以“生成式人工智能”领域为例

提出一种高校高价值专利技术机会识别方法,使用主题建模、突变级数法、机器学习与离群值检测算法,在评估出高校高价值专利的基础上,进一步识别出具有潜在技术机会的技术主题与专利技术。以“生成式人工智能”领域为例进行实证,研究结果表明:“生成式人工智能”领域的潜在技术主题集中在深度学习、神经网络与机器学习等前沿领域,AI影像、AI诊疗等技术为该领域的潜在技术机会,且上述技术均有国家相关政策大力支撑。本研究方法突破了单一技术机会识别方法识别结果针对性不强、识别专利价值不大、识别结果形式较为单一等核心问题,相关识别结果可以为高校技术转移、技术研发与技术创新提供决策支撑。

基于离群点检测和自适应参数的三支DBSCAN算法

针对经典的DBSCAN算法存在难以确定全局最优参数和误判离群点的问题,该算法首先从选择最优参数角度出发,通过数据集的分布特征生成Eps和MinPts列表,将两个列表中的参数进行全组合操作,把不同的参数组合依次进行聚类,从而寻找准确率最高点对应的参数。最后从离群点角度出发,将三支决策思想与离群点检测LOF算法进行结合。该算法与多种聚类算法进行效果对比分析,结果表明该算法能够全自动化选择全局最优参数,并提高聚类算法的准确性。

地铁盾构下穿高铁站房安全监测

针对地铁盾构下穿地表建筑物形变规律监测和复杂工况环境下地基SAR数据处理等问题,提出利用广义离群检验算法对地基SAR一阶差分相位数据进行粗差探测与剔除,联合地基SAR与测量机器人监测数据对地铁盾构下穿湘潭北站高铁站房进行形变监测与安全分析,并基于力学模型和Flac3D软件进行数值模拟,较好的得出盾构下穿高铁站房各阶段形变特征与规律。结果表明,广义离群检验算法有效解决了复杂工况环境下地基SAR数据遮挡问题,联合测量机器人数据得到与数值模型变形解算结果一致,且符合施工过程物理力学客观规律,基本实现盾构下穿地表建筑物安全监测,为类似地铁盾构安全施工检测提供参考。

基于改进K-means的局部离群点检测方法

离群点检测任务是指检测与正常数据在特征属性上存在显著差异的异常数据。大多数基于聚类的离群点检测方法主要从全局角度对数据集中的离群点进行检测,而对局部离群点的检测性能较弱。基于此,本文通过引入快速搜索和发现密度峰值方法改进K-means聚类算法,提出了一种名为KLOD(local outlier detection based on improved K-means and least-squares methods)的局部离群点检测方法,以实现对局部离群点的精确检测。首先,利用快速搜索和发现密度峰值方法计算数据点的局部密度和相对距离,并将二者相乘得到γ值。其次,将γ值降序排序,利用肘部法则选择γ值最大的k个数据点作为K-means聚类算法的初始聚类中心。然后,通过K-means聚类算法将数据集聚类成k个簇,计算数据点在每个维度上的目标函数值并进行升序排列。接着,确定数据点的每个维度的离散程度并选择适当的拟合函数和拟合点,通过最小二乘法对升序排列的每个簇的每1维目标函数值进行函数拟合并求导,以获取变化率。最后,结合信息熵,将每个数据点的每个维度目标函数值乘以相应的变化率进行加权,得到最终的异常得分,并将异常值得分较高的top-n个数据点视为离群点。通过人工数据集和UCI数据集,对KLOD、LOF和KNN方法在准确度上进行仿真实验对比。结果表明KLOD方法相较于KNN和LOF方法具有更高的准确度。本文提出的KLOD方法能够有效改善K-means聚类算法的聚类效果,并且在局部离群点检测方面具有较好的精度和性能。

改进DPC聚类算法的离群点检测与解释方法

为解决全局离群点检测方法无法对局部离群点进行检测,以及局部异常因子在面对大量局部离群点时性能下降的问题,利用k近邻(KNN)和核密度估计方法(KDE)提出一种基于改进快速搜索和发现密度峰值聚类算法(KDPC)的离群点检测与解释方法,该方法能够同时对数据点的全局和局部进行分析。首先,利用k近邻和核密度估计方法计算数据点的局部密度,代替传统DPC算法中根据截断距离计算的局部密度。其次,将数据点的k近邻距离之和作为全局异常值,并通过KDPC聚类算法计算簇密度以及数据点的局部异常值。最后,将数据点的全局与局部异常值进行乘积作为最终异常得分,选取异常得分最高的Top-n作为离群点,通过构建全局-局部异常值决策图对全局和局部离群点进行解释。利用人工数据集和UCI数据集进行实验并与10种常用离群点检测方法进行比较。结果表明,该方法对全局和局部离群点都有着较高的检测精度和检测性能,并且AUC方面受k值影响较小。同时,利用该方法对NBA球员数据进行分析讨论,进一步证明了该方法的实用性和有效性。

ECOD算法在飞机不稳定进近检测中的应用

在飞机进近和着陆阶段,一旦发生不稳定进近就可能导致航空事故发生,因此终端空域内不稳定进近检测是航空器运行监控领域的热点研究问题。针对终端区的不稳定进近检测,利用OpenSky提供的开源航空器监视数据提出了基于数据驱动的检测方法。从能量管理的角度入手,构建基于无监督异常检测(Empirical-Cumulative-Distribution-based Outlier Detection,ECOD)算法的不稳定进近检测模型,并结合主成分分析(Principal Components Analysis,PCA),获取了飞机能量状态的异常评分进而实现检测。复飞事件检测的验证分析结果表明检测模型在准确率与效率方面具有优势,模型可实现实时部署与在线更新。

基于D-S证据理论的无监督异常检测算法

在实际应用中,当数据集缺少真实标签或正常点数据量不足时,可能导致一分类支持向量机处于无监督情况。此外,当训练集中包含异常数据时,一分类支持向量机生成的决策边界将偏斜至异常数据区域。上述问题降低了异常数据的检测率,并导致分类器的性能变差。为了解决上述问题,基于K近邻算法将数据集分为可疑正常点数据集与可疑离群点数据集。其中,可疑正常点数据集用于一分类支持向量机训练与建模,对于可疑离群点数据集则采用D-S证据理论来识别其中的正常数据。实验结果表明:基于D-S理论的无监督异常检测算法可以有效地分离正常点与异常点,该算法在整体数据集上A_(uc)均值达到了0.83,且在可疑离群点数据集上A_(uc)均值达到了0.883。

基于三维重建外点剔除的船舶航迹自适应修正

为了保证船舶的安全性,通过三维重建外点剔除进行船舶航迹自适应修正。采用均值限差法对船舶三维点云数据进行滤波处理,通过变窗口方差比值方法优化均值限差法,提升船舶三维点云滤波精度。针对滤波处理后的船舶三维点云数据,结合八叉树自适应网格对船舶进行三维重建,更准确地拟合船舶的表面形状。由此,在屏幕上投影八叉树自适应网格生成投影多边形,检测其与用户确定的矩形包围盒间的相交性,定义与矩形包围盒不相交的八叉树自适应网格内的点为外点,并将其剔除,可以更准确地计算船舶航行,完成船舶航迹自适应修正。实验结果表明,所提方法的点云滤波总误差控制在1.5%以内,可有效抑制海面杂波等因素造成的外点影响,且规划的航线与原定航线一致性较高,可以保证船舶的运行安全。