基于Hadoop的VPN访问日志分析平台的研究与实现

Research and Implementation of VPN Access Log Analysis Platform Based on Hadoop

采用Hadoop分布式计算框架和数据仓库Hive构建一个日志分析平台,将VPN环境下不同系统或应用程序产生的多个相关的日志进行关联分析,以还原出通过VPN存取资源的轨迹行为,用以改善使用单机进行大量日志处理效率低下而且扩充性差的问题.平台产生的轨迹数据可以协助系统管理者找出数据外泄轨迹,了解资源是否被滥用,以及发现潜在的安全性威胁.

A platform is built for log analysis to process correlation analysis of multiple related log under different system or application program,which can restore the accessing resources behavior via VPN.The platform uses the Hadoop distributed computing framework and data warehouse Hive,which improves the efficiency of large number of log processing and poor extensibility of single host.The trail data generated can assist system administrator to identify data leakage path,which understands resources abuse and finds potential security threats.