基于Pig__Spark的分布式数据分析处理平台

Distributed data analysis and processing platform based on Pig__Spark

传统的数据分析平台Pig的执行引擎是MapReduce,由于MapReduce的局限性,使得数据处理过程中存在高延迟,内存开销大等缺点。为克服这些不足,文中基于当下最流行的内存计算框架Spark,在保留传统数据分析平台Pig语言特性和基础设施的基础上,开发实现了一种全新的数据分析处理平台,并通过具体实验对比两个数据平台的性能。实验结果证明,基于Saprk的数据分析平台在数据处理速度上远远高于传统的数据分析平台Pig。

The traditional data analysis platform Pig is developed based on MapReduce.Due to the limitations of MapReduce,Pig has some shortcoming,such as high latency and memory overhead in the process of data processing.In order to overcome these shortcomings,based on the most popular memory computing framework,this paper develops and implements a new data analysis and processing platform on the basis of Pig’s linguistic features and infrastructure.It compares the performance of the two data platform through the specific experiments.The experimental results show that the data analysis platform based on Saprk is faster than the traditional data analysis platform Pig in the data analysis and processing.