NetFlow流量数据的压缩存储与多维度聚合

The Compression Storage and Multi-dimensional Aggregation of NetFlow Traffic Data

基于NetFlow技术,实现网络流量数据的采集整理、压缩存储和多维聚合。数据采集采用全时段抽样采集,保证数据的准确和高效。针对数据的海量特点,提出了固定阈值和可变阈值两种数据压缩方法,大大降低了数据存储量。此外,针对不同的统计分析需求,提出了数据多维度聚合结构,涵盖了数据流中时间、协议、IP地址、端口等信息。最后应用于真实的流量数据进行统计分析,取得良好效果。

The paper accomplished the collection, compression storage and multi-dimensional aggregation of network traffic data on NetFlow. It ensured the accuracy and efficiency of data by making use of sample collection, and then, In allusion to the mass characteristic of network data, the paper proposed a fixed threshold and variable threshold data compression, greatly reduced the volume of data storage. In addition, according to different needs of statistical analysis, the paper put forward a multi-dimensional aggregation structure, covering time, protocol, IP address, port and other information in data streams. Finally, the real data streams were used in statistical analysis, and achieved good results.