面向实时水环境监测大数据分析的分布式处理框架研究

Research on distributed processing framework for big data analysis of real-time water environment monitoring

水质监测数据的迅速积累和增长,给水质研究带来了新的机遇和挑战。针对目前水质监测与分析面临的实时性差、数据显示不直观、数据处理效率低等问题,本文基于大数据技术和数据可视化技术,构建了用于实时水环境监测与大数据分析的分布式处理框架WaterSpark。使用改进的加拿大环境部长理事会水质指数(CCME-WQI)和Spark机器学习库(MLlib),进行云南省九大高原湖泊水质监测数据。结果表明,WaterSpark在实时水质传输、清洗归档、高效计算分析等方面表现出色,能够及时准确地捕获和分析大规模水质数据,分布式数据集和集群能够应对不断增长的水质数据,保证性能的可扩展性,支持更多的水质指标和更大规模的水质监测。

The rapid accumulation and growth of water quality monitoring data brings new opportunities and challenges to water quality research.In view of the problems of poor timeliness,unintuitive data display,and inefficient data processing faced by water quality monitoring and analysis at present,this paper constructs WaterSpark,a distributed processing framework for real-time water environment monitoring and big data analysis,based on big data technology and data visualization technology,using the improved Canadian Council of Ministers of the Environment water quality index(CCME-WQI) and the Spark machine learning library(MLlib),Applying the water quality monitoring data of nine plateau lakes in Yunnan province,the results show that WaterSpark has excellent performance in real-time water quality transmission,cleaning and archiving,and efficient computing and analysis.It can enable large-scale water quality data to be captured and analyzed timely and accurately,and the distributed data sets and clusters can cope with the growing water quality data to ensure performance scalability,and to support more water quality indicators and water quality monitoring on a larger scale.