This letter presents a new chunking method based on Maximum Entropy (ME) model with N-fold template correction model.First two types of machine learning models are described.Based on the analysis of the two models,the...This letter presents a new chunking method based on Maximum Entropy (ME) model with N-fold template correction model.First two types of machine learning models are described.Based on the analysis of the two models,then the chunking model which combines the profits of conditional probability model and rule based model is proposed.The selection of features and rule templates in the chunking model is discussed.Experimental results for the CoNLL-2000 corpus show that this approach achieves impressive accuracy in terms of the F-score:92.93%.Compared with the ME model and ME Markov model,the new chunking model achieves better performance.展开更多
Logs contain runtime information for both systems and users.As many of them use natural language,a typical log-based analysis needs to parse logs into the structured format first.Existing parsing approaches often take...Logs contain runtime information for both systems and users.As many of them use natural language,a typical log-based analysis needs to parse logs into the structured format first.Existing parsing approaches often take two steps.The first step is to find similar words(tokens)or sentences.Second,parsers extract log templates by replacing different tokens with variable placeholders.However,we observe that most parsers concentrate on precisely grouping similar tokens or logs.But they do not have a well-designed template extraction process,which leads to inconsistent accuracy on particular datasets.The root cause is the ambiguous definition of variable placeholders and similar templates.The consequences include abuse of variable placeholders,incorrectly divided templates,and an excessive number of templates over time.In this paper,we propose our online log parsing approach Cognition.It redefines variable placeholders via a strict lower bound to avoid ambiguity first.Then,it applies our template correction technique to merge and absorb similar templates.It eliminates the interference of commonly used parameters and thus isolates template quantity.Evaluation through 16 public datasets shows that Cognition has better accuracy and consistency than the state-of-the-art approaches.It also saves up to 52.1%of time cost on average than the others.展开更多
基金Supported by National Natural Science Foundation of China (No.60504021).
文摘This letter presents a new chunking method based on Maximum Entropy (ME) model with N-fold template correction model.First two types of machine learning models are described.Based on the analysis of the two models,then the chunking model which combines the profits of conditional probability model and rule based model is proposed.The selection of features and rule templates in the chunking model is discussed.Experimental results for the CoNLL-2000 corpus show that this approach achieves impressive accuracy in terms of the F-score:92.93%.Compared with the ME model and ME Markov model,the new chunking model achieves better performance.
基金supported by the National Key Research and Development Program of China under Grant No.2019YFB1802800the National Science Fund for Distinguished Young Scholars of China under Grant No.61725206。
文摘Logs contain runtime information for both systems and users.As many of them use natural language,a typical log-based analysis needs to parse logs into the structured format first.Existing parsing approaches often take two steps.The first step is to find similar words(tokens)or sentences.Second,parsers extract log templates by replacing different tokens with variable placeholders.However,we observe that most parsers concentrate on precisely grouping similar tokens or logs.But they do not have a well-designed template extraction process,which leads to inconsistent accuracy on particular datasets.The root cause is the ambiguous definition of variable placeholders and similar templates.The consequences include abuse of variable placeholders,incorrectly divided templates,and an excessive number of templates over time.In this paper,we propose our online log parsing approach Cognition.It redefines variable placeholders via a strict lower bound to avoid ambiguity first.Then,it applies our template correction technique to merge and absorb similar templates.It eliminates the interference of commonly used parameters and thus isolates template quantity.Evaluation through 16 public datasets shows that Cognition has better accuracy and consistency than the state-of-the-art approaches.It also saves up to 52.1%of time cost on average than the others.
