大数据清洗中的异常值检测与处理算法研究

Research on Outlier Detection and Processing Algorithm in Big Data Cleaning

在物联网环境异常值检测领域,基于支持向量机的方法能提高物联网环境中的数据质量,让物联网环境保持稳定。在实现过程中,通过数据预处理、特征选择、模型训练、参数优化等步骤,SVM模型在训练集和测试集上均表现出良好的性能,准确率、召回率、F1分数和AUC-ROC等指标均达到了较高的水平。文中提出了基于SVM检测结果的异常值处理策略,通过案例分析,验证了所提方法的有效性,并成功检测出物联网环境中的异常值,为保持物联网环境的稳定性提供了有力支持。

In the field of outlier detection in the Internet of Things environment,the method based on support vector machine can improve the data quality in the Internet of Things environment and keep the Internet of Things environment stable.In the implementation process,through data preprocessing,feature selection,model training,parameter optimization and other steps,the SVM model shows good performance on the training dataset and test set,and the accuracy,recall rate,F1 score and AUC-ROC indicators have reached a high level.This paper proposes an outlier processing strategy based on SVM detection results.Through case analysis,the effectiveness of the proposed method is verified,and the outliers in the Internet of Things environment are successfully detected,which provides strong support for maintaining the stability of the Internet of Things environment.