As software systems grow more and more complex,extensive techniques have been proposed to analyze the log data to obtain the insight of the system status.However,during log data analysis,tedious manual efforts are pai...As software systems grow more and more complex,extensive techniques have been proposed to analyze the log data to obtain the insight of the system status.However,during log data analysis,tedious manual efforts are paid to search interesting or informative log patterns from a huge volume of log data,named pattern-based queries.Although existing log management tools and DMBS systems can also support pattern-based queries,they suffer from a low efficiency.To deal with this problem,we propose a novel approach,named PLQ(Pattern-based Log Query).First,PLQ organizes logs into disjoint chunks and builds chunk-wise bitmap indexes for log types and attribute values.Then,based on bitmap indexes,PLQ finds candidate logs with a set of efficient bit-wise operations.Finally,PLQ fetches candidate logs and validates them according to the queried pattern.Extensive experiments are conducted on real-life datasets.According to experimental results,compared with existing log management systems,PLQ is more efficient in querying log patterns and has a higher pruning rate for filtering irrelevant logs.Moreover,in PLQ,since the ratio of the index size to the data size does not exceed 2.5%for log datasets of different sizes,PLQ has a high scalability.展开更多
基金This work was supported by the National Natural Science Foundation of China under Grant No.61672163the MIIT Project:Data Management Standards and Verfication for Industrial Internet Identifer Resoluation.
文摘As software systems grow more and more complex,extensive techniques have been proposed to analyze the log data to obtain the insight of the system status.However,during log data analysis,tedious manual efforts are paid to search interesting or informative log patterns from a huge volume of log data,named pattern-based queries.Although existing log management tools and DMBS systems can also support pattern-based queries,they suffer from a low efficiency.To deal with this problem,we propose a novel approach,named PLQ(Pattern-based Log Query).First,PLQ organizes logs into disjoint chunks and builds chunk-wise bitmap indexes for log types and attribute values.Then,based on bitmap indexes,PLQ finds candidate logs with a set of efficient bit-wise operations.Finally,PLQ fetches candidate logs and validates them according to the queried pattern.Extensive experiments are conducted on real-life datasets.According to experimental results,compared with existing log management systems,PLQ is more efficient in querying log patterns and has a higher pruning rate for filtering irrelevant logs.Moreover,in PLQ,since the ratio of the index size to the data size does not exceed 2.5%for log datasets of different sizes,PLQ has a high scalability.
