System logs record detailed information about system operation and areimportant for analyzing the system's operational status and performance. Rapidand accurate detection of system anomalies is of great significan...System logs record detailed information about system operation and areimportant for analyzing the system's operational status and performance. Rapidand accurate detection of system anomalies is of great significance to ensure system stability. However, large-scale distributed systems are becoming more andmore complex, and the number of system logs gradually increases, which bringschallenges to analyze system logs. Some recent studies show that logs can beunstable due to the evolution of log statements and noise introduced by log collection and parsing. Moreover, deep learning-based detection methods take a longtime to train models. Therefore, to reduce the computational cost and avoid loginstability we propose a new Word2Vec-based log unsupervised anomaly detection method (LogUAD). LogUAD does not require a log parsing step and takesoriginal log messages as input to avoid the noise. LogUAD uses Word2Vec togenerate word vectors and generates weighted log sequence feature vectors withTF-IDF to handle the evolution of log statements. At last, a computationally effi-cient unsupervised clustering is exploited to detect the anomaly. We conductedextensive experiments on the public dataset from Blue Gene/L (BGL). Experimental results show that the F1-score of LogUAD can be improved by 67.25%compared to LogCluster.展开更多
One particular challenge for large‑scale software systems is anomaly detection.System logs are a straightforward and common source of information for anomaly detection.Existing log‑based anomaly detectors are unusable...One particular challenge for large‑scale software systems is anomaly detection.System logs are a straightforward and common source of information for anomaly detection.Existing log‑based anomaly detectors are unusable in real‑world industrial systems due to high false‑positive rates.In this paper,we incorporate human feedback to adjust the detection model structure to reduce false positives.We apply our approach to two industrial large‑scale systems.Results have shown that our approach performs much better than state‑of‑the-art works with 50%higher accuracy.Besides,human feedback can reduce more than 70%of false positives and greatly improve detection precision.展开更多
基金funded by the Researchers Supporting Project No.(RSP.2021/102)King Saud University,Riyadh,Saudi ArabiaThis work was supported in part by the National Natural Science Foundation of China under Grant 61802030+2 种基金Natural Science Foundation of Hunan Province under Grant 2020JJ5602the Research Foundation of Education Bureau of Hunan Province under Grant 19B005the International Cooperative Project for“Double First-Class”,CSUST under Grant 2018IC24.
文摘System logs record detailed information about system operation and areimportant for analyzing the system's operational status and performance. Rapidand accurate detection of system anomalies is of great significance to ensure system stability. However, large-scale distributed systems are becoming more andmore complex, and the number of system logs gradually increases, which bringschallenges to analyze system logs. Some recent studies show that logs can beunstable due to the evolution of log statements and noise introduced by log collection and parsing. Moreover, deep learning-based detection methods take a longtime to train models. Therefore, to reduce the computational cost and avoid loginstability we propose a new Word2Vec-based log unsupervised anomaly detection method (LogUAD). LogUAD does not require a log parsing step and takesoriginal log messages as input to avoid the noise. LogUAD uses Word2Vec togenerate word vectors and generates weighted log sequence feature vectors withTF-IDF to handle the evolution of log statements. At last, a computationally effi-cient unsupervised clustering is exploited to detect the anomaly. We conductedextensive experiments on the public dataset from Blue Gene/L (BGL). Experimental results show that the F1-score of LogUAD can be improved by 67.25%compared to LogCluster.
基金ZTE Industry-University-Institute Cooperation Funds under Grant No.20200492.
文摘One particular challenge for large‑scale software systems is anomaly detection.System logs are a straightforward and common source of information for anomaly detection.Existing log‑based anomaly detectors are unusable in real‑world industrial systems due to high false‑positive rates.In this paper,we incorporate human feedback to adjust the detection model structure to reduce false positives.We apply our approach to two industrial large‑scale systems.Results have shown that our approach performs much better than state‑of‑the-art works with 50%higher accuracy.Besides,human feedback can reduce more than 70%of false positives and greatly improve detection precision.
