基于Hadoop与Spark的高校校园大数据平台研究

Research on University Campus Big Data Platform Based on Hadoop and Spark

对校园大数据分析是校园信息化发展的新思路。Hadoop是Apache基金会开发的分布式系统基础架构,它是集分布式计算、存储和管理为一体的生态系统。目前流行的Spark框架是与Hadoop生态系统中的MapReduce类似的一个分布式计算平台,Spark比MapReduce的速度更快且提供的功能更丰富。本文以数据采集、数据存储、数据分析、数据展现为主线,结合大数据领域最流行的Hadoop框架与Spark框架提出了高校校园大数据平台架构,详细阐述了架构各层次的具体功能,并对架构中关系数据库数据的采集存储进行了详细介绍,最后设计校园大数据分析原型系统来验证架构的可行性。

The analysis of campus big data is a new way of campus information development.Hadoop is a distributed system infrastructure developed by Apache Foundation,which is an ecosystem integrating distributed computing,storage and management.The current popular Spark framework is a distributed computing platform similar to MapReduce in the Hadoop ecosystem,and Spark is faster and more functional than MapReduce.With the main line of data collection,data storage,data analysis and data presentation,this paper puts forward the big data platform architecture of university campus in combination with the most popular Hadoop framework and Spark framework in big data fields,and expounds the specific functions of the architecture at all levels in detail,and gives a detailed description of the data collection and storage of the related coefficients in the architecture.Finally,the campus big data analysis prototype system is designed to verify the feasibility of the architecture.