An Efficient Way to Parse Logs Automatically for Multiline Events

In order to obtain information or discover knowledge from system logs,the first step is to performlog parsing,whereby unstructured raw logs can be transformed into a sequence of structured events.Although comprehensive studies on log parsing have been conducted in recent years,most assume that one event object corresponds to a single-line message.However,in a growing number of scenarios,one event object spans multiple lines in the log,for which parsing methods toward single-line events are not applicable.In order to address this problem,this paper proposes an automated log parsing method for multiline events(LPME).LPME finds multiline event objects via iterative scanning,driven by a set of heuristic rules derived from practice.The advantage of LPME is that it proposes a cohesion-based evaluation method for multiline events and a bottom-up search approach that eliminates the process of enumerating all combinations.We analyze the algorithmic complexity of LPME and validate it on four datasets from different backgrounds.Evaluations show that the actual time complexity of LPME parsing for multiline events is close to the constant time,which enables it to handle large-scale sample inputs.On the experimental datasets,the performance of LPME achieves 1.0 for recall,and the precision is generally higher than 0.9,which demonstrates the effectiveness of the proposed LPME.

ML-Parser:An Eficient and Accurate Online Log Parser

A log is a text message that is generated in various services,frameworks,and programs.The majority of log data mining tasks rely on log parsing as the first step,which transforms raw logs into formatted log templates.Existing log parsing approaches often fail to effectively handle the trade-off between parsing quality and performance.In view of this,in this paper,we present Multi-Layer Parser(ML-Parser),an online log parser that runs in a streaming manner.Specifically,we present a multi-layer structure in log parsing to strike a balance between efficiency and effectiveness.Coarse-grained tokenization and a fast similarity measure are applied for efficiency while fine-grained tokenization and an accurate similarity measure are used for effectiveness.In experiments,we compare ML-Parser with two existing online log parsing approaches,Drain and Spell,on ten real-world datasets,five labeled and five unlabeled.On the five labeled datasets,we use the proportion of correctly parsed logs to measure the accuracy,and ML-Parser achieves the highest accuracy on four datasets.On the whole ten datasets,we use Loss metric to measure the parsing quality.ML-Parse achieves the highest quality on seven out of the ten datasets while maintaining relatively high efficiency.

Cognition:Accurate and Consistent Linear Log Parsing Using Template Correction

Logs contain runtime information for both systems and users.As many of them use natural language,a typical log-based analysis needs to parse logs into the structured format first.Existing parsing approaches often take two steps.The first step is to find similar words(tokens)or sentences.Second,parsers extract log templates by replacing different tokens with variable placeholders.However,we observe that most parsers concentrate on precisely grouping similar tokens or logs.But they do not have a well-designed template extraction process,which leads to inconsistent accuracy on particular datasets.The root cause is the ambiguous definition of variable placeholders and similar templates.The consequences include abuse of variable placeholders,incorrectly divided templates,and an excessive number of templates over time.In this paper,we propose our online log parsing approach Cognition.It redefines variable placeholders via a strict lower bound to avoid ambiguity first.Then,it applies our template correction technique to merge and absorb similar templates.It eliminates the interference of commonly used parameters and thus isolates template quantity.Evaluation through 16 public datasets shows that Cognition has better accuracy and consistency than the state-of-the-art approaches.It also saves up to 52.1%of time cost on average than the others.