基于Hadoop的校园网站日志系统的设计与实现

Hadoop-based campus websites logging system design and implementation

每所高校校园网内成百上千的网站每天产生的Web访问日志量级都以千万计.利用大数据技术对这些海量、实时的数据进行集中搜集和长期存储,并通过后期大数据分析来挖掘这些日志的价值,无论从管理运维角度,还是从信息安全视角都具有实际意义.本文以上海交通大学为例,介绍了利用Hadoop大数据平台,存储并分析校园网内数千网站日志的系统设计和实现.

Web access logs daily produced from the hundreds of websites of each university campus network can be as much as tens of millions,to centrally collect and store these massive realtime and long-term data using big data technology and dig the value of these logs later using big data analysis have huge practical significance from the perspectives of both management of operation and maintenance and information security.This paper takes the Shanghai Jiao Tong University as an example and introduces Campus Logging System＇s design and implementation by storing and analyzing thousands of websites log using Hadoop big data platform.