Log anomaly detection is an important paradigm for system troubleshooting.Existing log anomaly detection based on Long Short-Term Memory(LSTM)networks is time-consuming to handle long sequences.Transformer model is in...Log anomaly detection is an important paradigm for system troubleshooting.Existing log anomaly detection based on Long Short-Term Memory(LSTM)networks is time-consuming to handle long sequences.Transformer model is introduced to promote efficiency.However,most existing Transformer-based log anomaly detection methods convert unstructured log messages into structured templates by log parsing,which introduces parsing errors.They only extract simple semantic feature,which ignores other features,and are generally supervised,relying on the amount of labeled data.To overcome the limitations of existing methods,this paper proposes a novel unsupervised log anomaly detection method based on multi-feature(UMFLog).UMFLog includes two sub-models to consider two kinds of features:semantic feature and statistical feature,respectively.UMFLog applies the log original content with detailed parameters instead of templates or template IDs to avoid log parsing errors.In the first sub-model,UMFLog uses Bidirectional Encoder Representations from Transformers(BERT)instead of random initialization to extract effective semantic feature,and an unsupervised hypersphere-based Transformer model to learn compact log sequence representations and obtain anomaly candidates.In the second sub-model,UMFLog exploits a statistical feature-based Variational Autoencoder(VAE)about word occurrence times to identify the final anomaly from anomaly candidates.Extensive experiments and evaluations are conducted on three real public log datasets.The results show that UMFLog significantly improves F1-scores compared to the state-of-the-art(SOTA)methods because of the multi-feature.展开更多
One particular challenge for large‑scale software systems is anomaly detection.System logs are a straightforward and common source of information for anomaly detection.Existing log‑based anomaly detectors are unusable...One particular challenge for large‑scale software systems is anomaly detection.System logs are a straightforward and common source of information for anomaly detection.Existing log‑based anomaly detectors are unusable in real‑world industrial systems due to high false‑positive rates.In this paper,we incorporate human feedback to adjust the detection model structure to reduce false positives.We apply our approach to two industrial large‑scale systems.Results have shown that our approach performs much better than state‑of‑the-art works with 50%higher accuracy.Besides,human feedback can reduce more than 70%of false positives and greatly improve detection precision.展开更多
Search logs in a timely and efficient manner are an important part of SRE (Site Reliability Engineer). Logs help us solve the problems during our development work. In this paper, we will introduce you a way how to bui...Search logs in a timely and efficient manner are an important part of SRE (Site Reliability Engineer). Logs help us solve the problems during our development work. In this paper, we will introduce you a way how to build an efficient logs analysis system based on kafka and Elastic Search. We hope you can learn something through the iteration of the Version and get some inspiration with your own log analysis system.展开更多
This paper presents a debugging system for multi-pole array acoustic logging （MPAL） tools. The debugging system proposed in this study can debug the MPAL tool system, sub-system and local electronics. In the test eq...This paper presents a debugging system for multi-pole array acoustic logging （MPAL） tools. The debugging system proposed in this study can debug the MPAL tool system, sub-system and local electronics. In the test equipment, we have used principal and subordinate structures, and interconnected the host computer and the front-end machine via Ethernet. The front-end machine is based on the ARM7 （advanced reduced instruction set computing （RISC） machine） technique, the processor of which runs an embedded operating system, namely, uClinux OS. We have analyzed the system telecommunication, human-machine interface circuit, transmitter mandrel interface circuit, receiver mandrel interface circuit, and board-level test interface circuit. The software used in the system consists of the embedded front-computer software and the host application software. We have explained in detail the flow chart of the boot loader in the embedded front-computer software. The host application software is composed of four application subroutines, which match with the functional modules of the system hardware. A net communication program based on the server^client mode is implemented by means of socket programming and multi-thread programming. Test results indicate that the data transmission rate of the system is higher than 1 MB/s, which completely meets the current requirements of the data transmission rate between the tool system and the wireline telemetry device. Application of the debugging system, which includes multiple level test methods, shows that the proposed system can fully meet the test requirements of MPAL at various levels.展开更多
In this paper we have developed a data logging and monitoring system, we validated the system by comparing the result from it with the existing one and found that the system performs slightly better than the existing ...In this paper we have developed a data logging and monitoring system, we validated the system by comparing the result from it with the existing one and found that the system performs slightly better than the existing work in the same area. This implies that the data logger and monitoring system is good and can be used to monitor solar energy variables even at the comfort of our homes. We fitted a model to the generated data and found that the meteorological variables considered accounted for 99.88% of the power output in the rainy seasons while 0.12% of the variation was not explained due to other factors. Solar panels inclined at an angle of 5° (Tilt) and facing South Pole perform optimally.展开更多
在基于三副本策略的分布式存储系统中,当存储节点上的硬盘出现故障时,常见的处理方式是等待系统预设的时间.如果该故障硬盘超时未恢复,才开始恢复故障硬盘上的副本.这种处理方式存在的问题是,当三副本组中存在故障副本时,如果该副本组...在基于三副本策略的分布式存储系统中,当存储节点上的硬盘出现故障时,常见的处理方式是等待系统预设的时间.如果该故障硬盘超时未恢复,才开始恢复故障硬盘上的副本.这种处理方式存在的问题是,当三副本组中存在故障副本时,如果该副本组再有一个副本所在的硬盘发生故障,将导致系统无法继续提供服务,且不能自动恢复.本文提出一种基于日志副本的改进的Raft共识算法,即LR-Raft (log replica based Raft),日志副本没有完整状态机,可以快速加入集群,并参与投票与共识,提升了存在故障硬盘时系统的可用性;可以解决短时间内三副本中两个副本故障导致集群不可用和丢失数据的问题.实验结果表明,在副本组中引入日志副本后,与原Raft相比,LR-Raft在不同的工作负载下读写时延均明显降低,吞吐量显著提升.展开更多
基金supported in part by the National Natural Science Foundation of China under Grant 62272062the Scientific Research Fund of Hunan Provincial Transportation Department(No.202143)the Open Fund ofKey Laboratory of Safety Control of Bridge Engineering,Ministry of Education(Changsha University of Science Technology)under Grant 21KB07.
文摘Log anomaly detection is an important paradigm for system troubleshooting.Existing log anomaly detection based on Long Short-Term Memory(LSTM)networks is time-consuming to handle long sequences.Transformer model is introduced to promote efficiency.However,most existing Transformer-based log anomaly detection methods convert unstructured log messages into structured templates by log parsing,which introduces parsing errors.They only extract simple semantic feature,which ignores other features,and are generally supervised,relying on the amount of labeled data.To overcome the limitations of existing methods,this paper proposes a novel unsupervised log anomaly detection method based on multi-feature(UMFLog).UMFLog includes two sub-models to consider two kinds of features:semantic feature and statistical feature,respectively.UMFLog applies the log original content with detailed parameters instead of templates or template IDs to avoid log parsing errors.In the first sub-model,UMFLog uses Bidirectional Encoder Representations from Transformers(BERT)instead of random initialization to extract effective semantic feature,and an unsupervised hypersphere-based Transformer model to learn compact log sequence representations and obtain anomaly candidates.In the second sub-model,UMFLog exploits a statistical feature-based Variational Autoencoder(VAE)about word occurrence times to identify the final anomaly from anomaly candidates.Extensive experiments and evaluations are conducted on three real public log datasets.The results show that UMFLog significantly improves F1-scores compared to the state-of-the-art(SOTA)methods because of the multi-feature.
基金ZTE Industry-University-Institute Cooperation Funds under Grant No.20200492.
文摘One particular challenge for large‑scale software systems is anomaly detection.System logs are a straightforward and common source of information for anomaly detection.Existing log‑based anomaly detectors are unusable in real‑world industrial systems due to high false‑positive rates.In this paper,we incorporate human feedback to adjust the detection model structure to reduce false positives.We apply our approach to two industrial large‑scale systems.Results have shown that our approach performs much better than state‑of‑the-art works with 50%higher accuracy.Besides,human feedback can reduce more than 70%of false positives and greatly improve detection precision.
文摘Search logs in a timely and efficient manner are an important part of SRE (Site Reliability Engineer). Logs help us solve the problems during our development work. In this paper, we will introduce you a way how to build an efficient logs analysis system based on kafka and Elastic Search. We hope you can learn something through the iteration of the Version and get some inspiration with your own log analysis system.
基金supported by National Science Foundation of China (61102102, 11134011, 11204380 and 11374371)Major National Science and Technology Projects (2011ZX05020-002)+2 种基金PetroChina Innovation Foundation (2014D-5006-0307)Science and Technology Project of CNPC (2014A-3912 and 2011B-4001)the Foundation of China University of Petroleum (KYJJ2012-05-07)
文摘This paper presents a debugging system for multi-pole array acoustic logging （MPAL） tools. The debugging system proposed in this study can debug the MPAL tool system, sub-system and local electronics. In the test equipment, we have used principal and subordinate structures, and interconnected the host computer and the front-end machine via Ethernet. The front-end machine is based on the ARM7 （advanced reduced instruction set computing （RISC） machine） technique, the processor of which runs an embedded operating system, namely, uClinux OS. We have analyzed the system telecommunication, human-machine interface circuit, transmitter mandrel interface circuit, receiver mandrel interface circuit, and board-level test interface circuit. The software used in the system consists of the embedded front-computer software and the host application software. We have explained in detail the flow chart of the boot loader in the embedded front-computer software. The host application software is composed of four application subroutines, which match with the functional modules of the system hardware. A net communication program based on the server^client mode is implemented by means of socket programming and multi-thread programming. Test results indicate that the data transmission rate of the system is higher than 1 MB/s, which completely meets the current requirements of the data transmission rate between the tool system and the wireline telemetry device. Application of the debugging system, which includes multiple level test methods, shows that the proposed system can fully meet the test requirements of MPAL at various levels.
文摘In this paper we have developed a data logging and monitoring system, we validated the system by comparing the result from it with the existing one and found that the system performs slightly better than the existing work in the same area. This implies that the data logger and monitoring system is good and can be used to monitor solar energy variables even at the comfort of our homes. We fitted a model to the generated data and found that the meteorological variables considered accounted for 99.88% of the power output in the rainy seasons while 0.12% of the variation was not explained due to other factors. Solar panels inclined at an angle of 5° (Tilt) and facing South Pole perform optimally.
文摘在基于三副本策略的分布式存储系统中,当存储节点上的硬盘出现故障时,常见的处理方式是等待系统预设的时间.如果该故障硬盘超时未恢复,才开始恢复故障硬盘上的副本.这种处理方式存在的问题是,当三副本组中存在故障副本时,如果该副本组再有一个副本所在的硬盘发生故障,将导致系统无法继续提供服务,且不能自动恢复.本文提出一种基于日志副本的改进的Raft共识算法,即LR-Raft (log replica based Raft),日志副本没有完整状态机,可以快速加入集群,并参与投票与共识,提升了存在故障硬盘时系统的可用性;可以解决短时间内三副本中两个副本故障导致集群不可用和丢失数据的问题.实验结果表明,在副本组中引入日志副本后,与原Raft相比,LR-Raft在不同的工作负载下读写时延均明显降低,吞吐量显著提升.
