期刊文献+

基于差分压缩的大规模日志压缩系统 被引量:2

Large-scale log compressing system based on differential compression
下载PDF
导出
摘要 大型信息系统的日志数据规模呈现快速增长趋势,导致线速压缩与存储大规模日志数据成为当今数据管理的一大挑战。对大量的网络系统日志进行了研究,发现日志数据存在冗余的结构模式,在内容上存在时间局部相似性。提出了基于模板的细粒度日志差分压缩架构,针对具体日志数据,可配置与其相适应的细粒度差分策略。实验结果表明,与gzip工具相比,所提日志压缩系统在压缩速度上提高了2~10倍,压缩率比gzip更低,可达到10%。 The scale of log data produced by the large scale information system is growing rapidly. It leads to the big challenge of line-speed compressing and saving the large scale log data. By analysis on massive network log data, it is found that the log data has redundant pattern in terms of log structure and time local similarity in terms of log content. A differential log compression architecture based on template is proposed. Fine-grained differential compressive strategies in the architecture can be configured for a special log data. Experimental results show that, compared with gizp, the proposed log compressing architecture improves 2~10 times' compressive speed and gain a better compressing ratio approaching to 10%.
出处 《通信学报》 EI CSCD 北大核心 2015年第S1期197-202,共6页 Journal on Communications
基金 中科院战略性先导科技专项基金资助项目(XDA06031000)~~
关键词 日志 差分压缩 细粒度 模板 log differential compression fine grain template
  • 相关文献

参考文献13

  • 1JANG J H,et al.Accelerating forex trading system through transaction log compression. So C Design Conference (ISOCC) 2014 International . 2014
  • 2SRIVASTAVA M,GARG,MISHRA P K.Analysis of data extraction and data cleaning in Web usage mining. Proceedings of the 2015International Conference on Advanced Research in Computer Science Engineering Technology (ICARCSET 2015) . 2015
  • 3DUMAIS S,et al.Understanding user behavior through log data and analysis. Ways of Knowing in HCI . 2014
  • 4LEB128. http://en.wikipedia.org/wiki/LEB128 . 2015
  • 5LONVICK C.The BSD Syslog Protocol. RFC 3164 .
  • 6CHRISTENSEN R.Improving compression of massive log data. http://www.erg.utal.edu . 2013
  • 7H?T?NEN K.et al.Comprehensive log compression with frequent patterns. Data Warehousing and Knowledge Discovery . 2003
  • 8DEOROWICZ S,GRABOWSKI S.Sub-atomic field processing for improved Web log compression. Proceedings of IEEE International Conference on Modern Problems of Radio Engineering,Telecommunications and Computer Science . 2008
  • 9DEOROWICZ S,GRABOWSKI S.Efficient preprocessing for Web log compression. International Journal of Computing . 2008
  • 10SKIBI?SKI P,SWACHA J.Fast and efficient log file compression. Proceedings of CEUR Workshop of 11th East-European Conference on Advances in Databases and Information Systems (ADBIS 2007) . 2007

二级参考文献4

  • 1Albitz P,Liu C.DNS and BIND[M].Sebastopol,USA:O'Reilly Media,1998.
  • 2Vixie P.BIND Software[EB/OL].(2009-03-16).https://www.isc.org/software/bind.
  • 3Mockapetris P.Domain Names Implementation and Specification[S].RFC 1035,1987.
  • 4Vixie P.Extension Mechanisms for DNS(EDNS0)[S].RFC 2671,1999.

共引文献4

同被引文献15

引证文献2

二级引证文献1

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部