A Study on Outlier Detection and Feature Engineering Strategies in Machine Learning for Heart Disease Prediction

This paper investigates the application ofmachine learning to develop a response model to cardiovascular problems and the use of AdaBoost which incorporates an application of Outlier Detection methodologies namely;Z-Score incorporated with GreyWolf Optimization(GWO)as well as Interquartile Range(IQR)coupled with Ant Colony Optimization(ACO).Using a performance index,it is shown that when compared with the Z-Score and GWO with AdaBoost,the IQR and ACO,with AdaBoost are not very accurate(89.0%vs.86.0%)and less discriminative(Area Under the Curve(AUC)score of 93.0%vs.91.0%).The Z-Score and GWO methods also outperformed the others in terms of precision,scoring 89.0%;and the recall was also found to be satisfactory,scoring 90.0%.Thus,the paper helps to reveal various specific benefits and drawbacks associated with different outlier detection and feature selection techniques,which can be important to consider in further improving various aspects of diagnostics in cardiovascular health.Collectively,these findings can enhance the knowledge of heart disease prediction and patient treatment using enhanced and innovativemachine learning(ML)techniques.These findings when combined improve patient therapy knowledge and cardiac disease prediction through the use of cutting-edge and improved machine learning approaches.This work lays the groundwork for more precise diagnosis models by highlighting the benefits of combining multiple optimization methodologies.Future studies should focus on maximizing patient outcomes and model efficacy through research on these combinations.

Changepoint Detection with Outliers Based on RWPCA

Changepoint detection faces challenges when outlier data are present. This paper proposes a multivariate changepoint detection method which is based on the robust WPCA projection direction and the robust RFPOP method, RWPCA-RFPOP method. Our method is double robust which is suitable for detecting mean changepoints in multivariate normal data with high correlations between variables that include outliers. Simulation results demonstrate that our method provides strong guarantees on both the number and location of changepoints in the presence of outliers. Finally, our method is well applied in an ACGH dataset.

A Study of Detection of Outliers for Working and Non-Working Days Air Quality in Kolkata, India: A Case Study

A variety of factors affect air quality, making it a difficult issue. The level of clean air in a certain area is referred to as air quality. It is challenging for conventional approaches to correctly discover aberrant values or outliers due to the significant fluctuation of this sort of data, which is influenced by Climate change and the environment. With accelerating industrial expansion and rising population density in Kolkata City, air pollution is continuously rising. This study involves two phases, in the first phase imputation of missing values and second detection of outliers using Statistical Process Control (SPC), and Functional Data Analysis (FDA), studies to achieve the efficacy of the outlier identification methodology proposed with working days and Nonworking days of the variables NO2, SO2, and O3, which were used for a year in a row in Kolkata, India. The results show how the functional data approach outshines traditional outlier detection methods. The outcomes show that functional data analysis vibrates more than the other two approaches after imputation, and the suggested outlier detector is absolutely appropriate for the precise detection of outliers in highly variable data.

CLOF Based Outlier Detection Algorithm of Temperature Data for Ethylene Cracking Furnace

The flue temperature is one of the important indicators to characterize the combustion state of an ethylene cracker furnace,the outliers of temperature data can lead to the false alarm.Conventional outlier detection algorithms such as the Isolation Forest algorithm and 3-sigma principle cannot detect the outliers accurately.In order to improve the detection accuracy and reduce the computational complexity,an outlier detection algorithm for flue temperature data based on the CLOF(Clipping Local Outlier Factor,CLOF)algorithm is proposed.The algorithm preprocesses the normalized data using the cluster pruning algorithm,and realizes the high accuracy and high efficiency outlier detection in the outliers candidate set.Using the flue temperature data of an ethylene cracking furnace in a petrochemical plant,the main parameters of the CLOF algorithm are selected according to the experimental results,and the outlier detection effect of the Isolation Forest algorithm,the 3-sigma principle,the conventional LOF algorithm and the CLOF algorithm are compared and analyzed.The results show that the appropriate clipping coefficient in the CLOF algorithm can significantly improve the detection efficiency and detection accuracy.Compared with the outlier detection results of the Isolation Forest algorithm and 3-sigma principle,the accuracy of the CLOF detection results is increased,and the amount of data calculation is significantly reduced.

A Novel Outlier Detection with Feature Selection Enabled Streaming Data Classification

Due to the advancements in information technologies,massive quantity of data is being produced by social media,smartphones,and sensor devices.The investigation of data stream by the use of machine learning(ML)approaches to address regression,prediction,and classification problems have received consid-erable interest.At the same time,the detection of anomalies or outliers and feature selection(FS)processes becomes important.This study develops an outlier detec-tion with feature selection technique for streaming data classification,named ODFST-SDC technique.Initially,streaming data is pre-processed in two ways namely categorical encoding and null value removal.In addition,Local Correla-tion Integral(LOCI)is used which is significant in the detection and removal of outliers.Besides,red deer algorithm(RDA)based FS approach is employed to derive an optimal subset of features.Finally,kernel extreme learning machine(KELM)classifier is used for streaming data classification.The design of LOCI based outlier detection and RDA based FS shows the novelty of the work.In order to assess the classification outcomes of the ODFST-SDC technique,a series of simulations were performed using three benchmark datasets.The experimental results reported the promising outcomes of the ODFST-SDC technique over the recent approaches.

Outliers rejection in similar image matching

Background Image matching is crucial in numerous computer vision tasks such as 3D reconstruction and simultaneous visual localization and mapping.The accuracy of the matching significantly impacted subsequent studies.Because of their local similarity,when image pairs contain comparable patterns but feature pairs are positioned differently,incorrect recognition can occur as global motion consistency is disregarded.Methods This study proposes an image-matching filtering algorithm based on global motion consistency.It can be used as a subsequent matching filter for the initial matching results generated by other matching algorithms based on the principle of motion smoothness.A particular matching algorithm can first be used to perform the initial matching;then,the rotation and movement information of the global feature vectors are combined to effectively identify outlier matches.The principle is that if the matching result is accurate,the feature vectors formed by any matched point should have similar rotation angles and moving distances.Thus,global motion direction and global motion distance consistencies were used to reject outliers caused by similar patterns in different locations.Results Four datasets were used to test the effectiveness of the proposed method.Three datasets with similar patterns in different locations were used to test the results for similar images that could easily be incorrectly matched by other algorithms,and one commonly used dataset was used to test the results for the general image-matching problem.The experimental results suggest that the proposed method is more accurate than other state-of-the-art algorithms in identifying mismatches in the initial matching set.Conclusions The proposed outlier rejection matching method can significantly improve the matching accuracy for similar images with locally similar feature pairs in different locations and can provide more accurate matching results for subsequent computer vision tasks.

Outlier Detection of Air Quality for Two Indian Urban Cities Using Functional Data Analysis

Human living would be impossible without air quality. Consistent advancements in practically every aspect of contemporary human life have harmed air quality. Everyday industrial, transportation, and home activities turn up dangerous contaminants in our surroundings. This study investigated two years’ worth of air quality and outlier detection data from two Indian cities. Studies on air pollution have used numerous types of methodologies, with various gases being seen as a vector whose components include gas concentration values for each observation per-formed. We use curves to represent the monthly average of daily gas emissions in our technique. The approach, which is based on functional depth, was used to find outliers in the city of Delhi and Kolkata’s gas emissions, and the outcomes were compared to those from the traditional method. In the evaluation and comparison of these models’ performances, the functional approach model studied well.

基于期望核密度离群因子的离群点检测算法

针对基于密度的离群点检测方法在不同分布的数据集上检测精度低的问题,提出了一种基于期望核密度离群因子的离群点检测算法。首先,引入k近邻和反向k近邻扩展邻域空间(ENS)代替传统的k邻域范围,更加全面地考虑数据对象的邻域信息;其次,在传统核密度估计(KDE)方法的基础上引入多元高斯函数,在扩展邻域空间内估计数据对象的密度,同时借鉴自适应核带宽的思想,更好地适应不同数据集的数据分布;然后,给出期望距离的概念,进一步区分局部离群点和位于低密度区域的正常点;最后,定义了期望核密度离群因子刻画数据对象离群程度。在人工数据集和真实数据集上对所提算法进行实验验证,并与部分传统算法进行对比,验证了所提算法的有效性。

基于贡献度和数据有效性检验的共识机制

将区块链技术引入到分布式数据维护系统,旨在解决基于传统中心化数据库的分布式系统存在的数据维护不透明、数据易被篡改、历史记录不可追溯等问题,提出一种基于贡献度和数据有效性检验的共识机制.该算法提出一种贡献度优先的随机可验证领导者选举机制,保证记账权分配的随机性及可验证性.进一步引入密度峰值算法对交易数据有效性进行校验,对打包区块的正确性达成共识.最后将所提出的共识机制应用于梅花鹿分布式养殖场场景,结果验证了密度峰值算法在交易数据有效性检测任务中的准确性和高效性.出块时延分析和安全性分析表明,所提出的共识机制能够满足数据有效性验证的实时性需求,能耗较小,具有很强的灾备能力.

基于5G移动通信技术在医疗设备管理中的研究

目的 为解决现有医疗设备通信技术滞后、设备管理能力欠佳等问题,提出一种基于5G移动通信技术在医疗设备管理中的方法,以提高医疗设备之间的数据信息通信能力。方法 基于5G通信技术,融合安全传输层协议,对传输数据进行加密;结合WebRTC技术,实现远程视频通话和会诊;使用离群检测算法对数据进行处理,提高通信数据的信息检索能力;加入网络波动检测模块,降低在远程访问中网络波动带来的影响。结果 通过记录某医疗机构某年1—10月的远程视频次数及网络波动次数和触发网络加速模块的次数可知,本系统将目标特征误差保持在2%以下,在系统运行过程中非常稳定且适用。结论 本系统大大减轻了系统主机的压力,具有较高的工作效率及较低的误差率和波动率,可达到远程医用和远程急救的要求。

基于映射距离比离群因子的离群点检测算法

针对基于邻近性的离群点检测方法需要花费大量时间过滤正常点,并且在检测全局离群点时难以检测出局部离群点的问题,提出一种基于映射距离比离群因子离群点检测(MDROF)算法。首先,为了减少正常点在检测过程中的时间消耗,给出了差异相似度的概念,通过定义差异相似度剪枝因子过滤掉数据集中的大部分正常点。其次,定义映射k距离,通过映射距离与可达距离的比值刻画数据对象的局部离群程度,通过可达密度刻画数据对象的全局离群程度。最后,结合数据对象相互近邻点的平均排位定义映射距离比离群因子来检测离群点。在人工数据集以及真实数据集上分别对该算法与其他经典的离群点检测算法在精确率、AUC值和离群点发现曲线上进行实验对比分析。实验结果证明MDROF算法在离群点检测的准确性和稳定性上明显优于对比算法。

基于自适应高斯渐进滤波的工程车GNSS/INS紧组合定位

研究了量测野值影响下的工程车GNSS/INS紧组合定位问题,提出了一种基于自适应高斯渐进滤波的车辆定位方法。首先,为降低量测野值对滤波器的破坏风险,利用假设检验方法对量测野值进行检测和剔除;其次,对于野值漏检测引起的定位性能下降的问题,设计了自适应的高斯渐进滤波方法来补偿量测的不确定性;特别地,利用线性化误差与系统估计误差的变化关系,对渐进量测更新方式进行了改进,从而实现对线性化误差的间接补偿。最后,通过工程车GNSS/INS紧组合定位实验进行结果分析,验证了所提方法的可靠性和优越性。

启发式k-means聚类算法的改进研究

启发式k-means聚类算法通过在k-means第一次迭代后查看附近的集群来预测每个数据点可能会被划分到的集群子集,有效地加快了算法的运行速度。但由于启发式算法存在随机选择初始聚类中心以及无法有效识别数据集中离群点的缺陷,导致聚类结果的误差平方和较大并且轮廓系数偏小。针对这一问题,提出了CHk-means算法,该算法引入仔细播种方法,克服了启发式k-means算法随机选择初始聚类中心带来的局部最优解问题;该算法引入局部异常因子LOF算法对离群点进行检测,降低了离群点数据对聚类结果的影响。在多个数据集上对3种算法进行对比试验,结果表明CHk-means算法可有效降低聚类结果的误差平方和,增强聚类的轮廓系数,使聚类质量得到明显改善。

颠覆性技术识别与扩散趋势预测:概念模型与实证分析

发现并判断技术颠覆性潜力和扩散趋势,能够为国家和政府科技资源分配与未来产业的超前布局提供精准决策依据。首先,本文构建了颠覆性技术识别与扩散趋势预测的概念模型;其次,依据此模型以3D打印领域为例,从离群性和影响力维度识别颠覆性专利,提取颠覆性技术;最后,基于识别的颠覆性专利的施引专利,将自动标签和战略坐标应用于技术主题扩散路径绘制中,提出一种新的多位态自动标签技术主题扩散趋势预测方法,用于揭示核心、边缘、成熟、新兴等位态主题之间的动态扩散关系。研究发现,离群专利与颠覆性技术之间存在共生、匹配和关联的内在逻辑关系,从离群专利视角识别颠覆性技术具有可行性;1955—2017年,3D打印领域的颠覆性技术主要分布在高端装备制造、生物医药和材料3大方向,突出的技术领域是运输、发动机/泵/涡轮机、生物材料分析、半导体、环境技术;多位态自动标签技术主题扩散趋势预测结果显示,生物医疗3D打印技术主题未来发展潜力巨大。

高校高价值专利技术机会识别研究——以“生成式人工智能”领域为例

提出一种高校高价值专利技术机会识别方法,使用主题建模、突变级数法、机器学习与离群值检测算法,在评估出高校高价值专利的基础上,进一步识别出具有潜在技术机会的技术主题与专利技术。以“生成式人工智能”领域为例进行实证,研究结果表明:“生成式人工智能”领域的潜在技术主题集中在深度学习、神经网络与机器学习等前沿领域,AI影像、AI诊疗等技术为该领域的潜在技术机会,且上述技术均有国家相关政策大力支撑。本研究方法突破了单一技术机会识别方法识别结果针对性不强、识别专利价值不大、识别结果形式较为单一等核心问题,相关识别结果可以为高校技术转移、技术研发与技术创新提供决策支撑。

基于离群点检测和自适应参数的三支DBSCAN算法

针对经典的DBSCAN算法存在难以确定全局最优参数和误判离群点的问题,该算法首先从选择最优参数角度出发,通过数据集的分布特征生成Eps和MinPts列表,将两个列表中的参数进行全组合操作,把不同的参数组合依次进行聚类,从而寻找准确率最高点对应的参数。最后从离群点角度出发,将三支决策思想与离群点检测LOF算法进行结合。该算法与多种聚类算法进行效果对比分析,结果表明该算法能够全自动化选择全局最优参数,并提高聚类算法的准确性。

基于CART决策树的分布式数据离群点检测算法

在分布式计算环境中,离群点通常表示数据中的异常情况,例如故障、欺诈、攻击等。通过检测分布式数据的离群点,可以对这些异常数据进行集中处理,保护系统和数据的安全。而进行离群点检测时,不仅要考虑数据的规模和复杂性,还要在分布式环境下高效地发现离群点。因此,提出一种基于CART决策树的分布式数据离群点检测算法。在构建CART决策树时,使用类间中心距离作为分裂准则,根据分离类别对训练数据进行分类,从而确定数据的类型。在上述基础上,考虑到离群点的分布模式与其周围数据对象不同,使用空间局部偏离因子(SLDF)对空间内各个数据对象之间的离群程度展开度量,同时在高维空间内展开网格划分,引入SLDF算法检测剩余离群点集,最终实现分布式数据离群点检测。实验结果表明,所提方法的离散点检测错误率在0.010以内,可以更加精准地实现分布式数据离群点检测,具有良好的检测性能。

地铁盾构下穿高铁站房安全监测

针对地铁盾构下穿地表建筑物形变规律监测和复杂工况环境下地基SAR数据处理等问题,提出利用广义离群检验算法对地基SAR一阶差分相位数据进行粗差探测与剔除,联合地基SAR与测量机器人监测数据对地铁盾构下穿湘潭北站高铁站房进行形变监测与安全分析,并基于力学模型和Flac3D软件进行数值模拟,较好的得出盾构下穿高铁站房各阶段形变特征与规律。结果表明,广义离群检验算法有效解决了复杂工况环境下地基SAR数据遮挡问题,联合测量机器人数据得到与数值模型变形解算结果一致,且符合施工过程物理力学客观规律,基本实现盾构下穿地表建筑物安全监测,为类似地铁盾构安全施工检测提供参考。

基于多元离群点检测的动态目标去除SLAM方法

考虑动态环境下的目标移动对同步定位与建图(SLAM)位姿估计精度的影响,提出一种通过稠密光流计算像素运动并经过离群点检测的动态目标SLAM算法。采用稠密光流法计算图像序列的每个像素的运动信息进行动态目标判断,利用离群点检测对动态目标进行提取,通过均值滤波对动态目标进行模糊剔除,消除动态目标对SLAM精度的影响。在TUM数据集与定制数据集上进行实验,在TUM数据集测试中,与基于特征点法的Orb-slam3标杆算法进行对比分析,在动态目标影响条件下,该算法得到的轨迹误差降低43.25%;搭建开放式四旋翼无人机测试系统,在定制数据集中,进行飞行试验,得到的估计轨迹位置误差控制在1 m内,满足使用场景要求,进一步验证了算法的有效性。

基于改进K-means的局部离群点检测方法

离群点检测任务是指检测与正常数据在特征属性上存在显著差异的异常数据。大多数基于聚类的离群点检测方法主要从全局角度对数据集中的离群点进行检测,而对局部离群点的检测性能较弱。基于此,本文通过引入快速搜索和发现密度峰值方法改进K-means聚类算法,提出了一种名为KLOD(local outlier detection based on improved K-means and least-squares methods)的局部离群点检测方法,以实现对局部离群点的精确检测。首先,利用快速搜索和发现密度峰值方法计算数据点的局部密度和相对距离,并将二者相乘得到γ值。其次,将γ值降序排序,利用肘部法则选择γ值最大的k个数据点作为K-means聚类算法的初始聚类中心。然后,通过K-means聚类算法将数据集聚类成k个簇,计算数据点在每个维度上的目标函数值并进行升序排列。接着,确定数据点的每个维度的离散程度并选择适当的拟合函数和拟合点,通过最小二乘法对升序排列的每个簇的每1维目标函数值进行函数拟合并求导,以获取变化率。最后,结合信息熵,将每个数据点的每个维度目标函数值乘以相应的变化率进行加权,得到最终的异常得分,并将异常值得分较高的top-n个数据点视为离群点。通过人工数据集和UCI数据集,对KLOD、LOF和KNN方法在准确度上进行仿真实验对比。结果表明KLOD方法相较于KNN和LOF方法具有更高的准确度。本文提出的KLOD方法能够有效改善K-means聚类算法的聚类效果,并且在局部离群点检测方面具有较好的精度和性能。