摘要
为了处理大量分布式存储的农业环境数据,提高农业生产效率,对高斯混合模型聚类算法进行了改进,提出了一种基于分布式聚类的农业环境数据异常检测方法.在Spark分布式计算框架下,首先对数据进行粗聚类,得到初始化模型;然后利用Spark迭代更新模型直至稳定,其中Map阶段将样本点分配到模型,Reduce阶段更新模型个数及参数;最后利用聚类结果,实现环境异常值的检测.实验结果表明该方法可行有效.
In order to process the massive agricultural environmental data stored in distributed system and improve the production efficiency, the clustering algorithm based on Gaussian Mixture Model (GMM) is modified in this paper. Based on this, an environmental anomaly detection method during crop growth is proposed. Under the Spark distributed computing framework, firstly, a pre-clustering algorithm is adopted to initialize the models. Secondly, Spark is utilized to update the models iterationally until it gets stable. In each iteration, Map phase distributes sample points to the models, Reduce phase renews the numbers of models and parameters. Finally, the detection of environmental anomaly is completed by taking advantages of the clustering result. The experimental results show that this approach is practically feasible and effective.
作者
余玥
邓丽
庞洪霖
费敏锐
YU Yue;DENG Li;PANG Hong-lin;FEI Min-rui(School of Mechatronics Engineering and Automation,Shanghai University,Shanghai 200072,Chin;Shanghai Key Laboratory of Power Station Automation Technology,Shanghai 200072,China)
出处
《应用科学学报》
CAS
CSCD
北大核心
2018年第6期1010-1021,共12页
Journal of Applied Sciences
基金
上海市科委重点项目基金(No.14DZ1206302)资助
关键词
高斯混合模型聚类
农业环境数据
异常检测
SPARK
Gaussian mixture model (GMM) clustering
agriculture environmental data
anomaly detection
Spark