摘要
随着近几年国内信息化水平不断提高,国网浙江电力主机、网络等设备数量急剧上升,业务系统的访问和操作量也迅速增加,传统的日志审计系统在应对超大规模海量日志的存储和分析时能力明显不足。针对海量日志采集、处理的需求,结合国网浙江省电力公司的实际情况,提出了一种基于开源ElasticSearch(弹性搜索)和Logstash(日志采集)的统一日志分析系统,并整合强隔离装置、toprowMQ消息队列、Kafka消息队列、Spark Streaming流数据处理框架,保证数据的安全性、可靠性和实时性。最后,通过日志分析系统在国网浙江电力的应用情况,验证了该系统的功能。
With the improvement of domestic informatization level in recent years, the number of hosts and network equipment in Zhejiang Electric Power Company increases rapidly, so do the system access and operations. Therefore, the traditional log audit system is difficult to store and analyze large-scale mass log. In order to meet the needs of log collection and dealing, a unified log analysis system based on ElasticSearch and Logstash is proposed in accordance with practical situation of Zhejiang Electric Power Company, which combines strong isolation devices, toprowMQ, Kafka cluster and Spark Streaming to guarantee the data are safe,reliable and real-time. Finally, the system functions are verified by the application of the log analysis system in Zhejiang Electric Power Company.
出处
《浙江电力》
2017年第12期27-32,共6页
Zhejiang Electric Power
