基于Spark的并行K-means气象数据挖掘研究

Parallel K-means algorithm based on spark in meteorological data

随着气象监测水平的不断提高,气象数据在短时间内可大量收集,充分从海量的气象数据中提取有价值的信息,是气象研究的关键所在。气象数据资源极其丰富,增长迅速,需要大量的科学计算,传统计算在大规模数据处理中需较长时间,无法满足数据即时转化的需求,大数据平台并行化处理可有效解决此问题。文中提出并行K-means算法对气候区进行划分方法,应用Spark平台进行处理,能快速有效地分析数据并从中发现有用信息,对气象领域研究具有重要的现实意义。

As the weather improved continuously monitoring meteorological data in a short time to collect a large number of fully tap extracting valuable information from vast amounts of meteorological data,meteorological forecast and severe weather warning key. Meteorological data resources are extremely rich,rapid growth requires a lot of scientific computing,the conventional computing take longer in large-scale data processing,the data can not meet the immediate needs of the conversion. Big Data platform parallel processing can effectively solve this problem. This paper uses Spark platform for massive data processing advantages,proposes a parallel K-means algorithm,which can effectively analyze the data and discover useful information,has important practical implications for the field of meteorology.