基于MapReduce的气候数据的分析的设计与实现

在社会的发展过程中,天气对社会中的很多行业都有一定的影响。在当前阶段,天气预报作为全球主要对天气进行预测的手段,在这种情况下,全球的气候行业也产生了大量的数据(PB级别),并且数据每年还在持续的增长。为了解决使用传统的关系型数据库很难存储与分析的问题,需要采用大数据的相关技术对气候数据进行分析。本设计采用MapReduce生态圈的一系列的工具,通过使用虚拟机的平台,采用爬虫软件进行数据爬取的方式,用Map阶段和Reduce阶段的数据分析,分布式系统的数据存储的方法,实现气象数据的采集分析,并且将其展现出来的方式展示。使用这系列流程的系统可以辅助有关部门或企业进行决策部署。

In the process of social development,weather has a certain impact on many industries in the society.At the current stage,weather forecasting is the main global weather forecasting tool,and in this context,the global climate industry also produces a large amount of data(petabytes),and the data continues to grow every year.In order to solve the problem that using traditional relational database is difficult to store and analyze,it is necessary to use big data-related technologies to analyze climate data.This design adopts a series of tools of MapReduce ecosystem.Through the use of virtual machine platform,crawler software is adopted for data crawling,data analysis in Map stage and Reduce stage,and data storage in distributed system,meteorological data collection and analysis are realized and displayed.Systems that use this series of processes can assist departments or organizations in their decision making and deployment.