基于大数据技术的服务器日志采集分析方法

A log collection and analysis method based on big data technology

对大数据量日志采用了一种新型的高可靠性、分布式的聚合传输信息收集Flume技术,同时结合无核相关向量机算法对服务器采集日志进行主机资源利用率的分析预测。通过本模型,实现了从海量日志中挖掘提取出高价值信息,有效地对服务器进行资源的合理化配置。验证结果表明Flume技术结合无核相关向量机算法在服务器日志采集分析上的可行性、有效性。

This paper uses a new type of high reliability and distributed aggregation technology to collect Flume technology, and combines the non-kernel correlation vector machine algorithm to analyze and forecast the host resource utilization of the server collection log. Through this model, the high value information can be extracted from the massive log, and the rationalization of the resource is effectively allocated. The results show that Flume technology is feasible and effective in the analysis of server log collection by combined with the non-kernel correlation vector machine algorithm.