基于大数据技术的数据分析处理平台设计与实现

Design and Implementation of Data Analysis and Processing Platform Based on Big Data Technology

针对目前大数据分析处理组件多而乱、平台技术架构复杂等问题,综合应用MapReduce、Hive、Sqoop等大数据技术,结合Scrapy数据采集框架,设计一种数据分析处理平台,利用SpringBoot及Vue技术搭建Web系统,实现处理结果的可视化。应用该平台对中国蔬菜网上的某一批发市场数据进行分析处理,得出年度蔬菜最高最低差价、年度价格排名前十的蔬菜品种等指标统计结果,分析结果对该批发市场的蔬菜投放及价格调控具有一定的指导意义。

In view of the problems such as multiple and chaotic components of big data analysis and processing and complex platform technical architecture,this paper comprehensively applies big data technologies such as MapReduce,Hive and Sqoop,combined with Scrapy data acquisition framework,to design a data analysis and processing platform,and uses SpringBoot and Vue technologies to build a Web system.It realizes the visualization of processing results.In this paper,the platform is used to analyze and process the data of a wholesale market in China Vegetable Net,and the highest and lowest price difference of annual vegetables and the top ten vegetable varieties in annual price are obtained.The results of the analysis have a certain guiding significance for the vegetable supply and price control in the wholesale market.