AgamottoEye:Recovering Request Flow for Cloud Systems via Log Analysis

Cloud applications are implemented on top of different distributed systems to provide online service.A service request is decomposed into multiple sub-tasks,which are dispatched to different distributed systems components.For cloud providers,monitoring the execution of a service request is crucial to promptly find problems that may compromise cloud availability.In this paper,we present AgamottoEye,to automatically construct request flow from existing logs.AgamottoEye addresses the challenges of analyzing interleaved log instances,and can successfully extract request flow spread across multiple distributed systems.Our experiments with Hadoop2/YARN show that AgamottoEye can analyze 25,050 log instances in 57.4s,and the extracted request flow information is helpful with error detection and diagnosis.

Log Anomaly Detection Based on Hierarchical Graph Neural Network and Label Contrastive Coding

System logs are essential for detecting anomalies,querying faults,and tracing attacks.Because of the time-consuming and labor-intensive nature of manual system troubleshooting and anomaly detection,it cannot meet the actual needs.The implementation of automated log anomaly detection is a topic that demands urgent research.However,the prior work on processing log data is mainly one-dimensional and cannot profoundly learn the complex associations in log data.Meanwhile,there is a lack of attention to the utilization of log labels and usually relies on a large number of labels for detection.This paper proposes a novel and practical detection model named LCC-HGLog,the core of which is the conversion of log anomaly detection into a graph classification problem.Semantic temporal graphs(STG)are constructed by extracting the raw logs’execution sequences and template semantics.Then a unique graph classifier is used to better comprehend each STG’s semantic,sequential,and structural features.The classification model is trained jointly by graph classification loss and label contrastive loss.While achieving discriminability at the class-level,it increases the fine-grained identification at the instance-level,thus achieving detection performance even with a small amount of labeled data.We have conducted numerous experiments on real log datasets,showing that the proposed model outperforms the baseline methods and obtains the best all-around performance.Moreover,the detection performance degrades to less than 1%when only 10%of the labeled data is used.With 200 labeled samples,we can achieve the same or better detection results than the baseline methods.

An Efficient Way to Parse Logs Automatically for Multiline Events

In order to obtain information or discover knowledge from system logs,the first step is to performlog parsing,whereby unstructured raw logs can be transformed into a sequence of structured events.Although comprehensive studies on log parsing have been conducted in recent years,most assume that one event object corresponds to a single-line message.However,in a growing number of scenarios,one event object spans multiple lines in the log,for which parsing methods toward single-line events are not applicable.In order to address this problem,this paper proposes an automated log parsing method for multiline events(LPME).LPME finds multiline event objects via iterative scanning,driven by a set of heuristic rules derived from practice.The advantage of LPME is that it proposes a cohesion-based evaluation method for multiline events and a bottom-up search approach that eliminates the process of enumerating all combinations.We analyze the algorithmic complexity of LPME and validate it on four datasets from different backgrounds.Evaluations show that the actual time complexity of LPME parsing for multiline events is close to the constant time,which enables it to handle large-scale sample inputs.On the experimental datasets,the performance of LPME achieves 1.0 for recall,and the precision is generally higher than 0.9,which demonstrates the effectiveness of the proposed LPME.

A Survey on Event Mining for ICT Network Infrastructure Management

1 IntroductionNowadays in China, there are more than six hundred million netizens [1]. On April 11, 2015, the nmnbet of simultaneous online users of the Chinese instant message application QQ reached two hundred million [2]. The fast growth ol the lnternet pusnes me rapid development of information technology （IT） and communication technology （CT）. Many traditional IT service and CT equipment providers are facing the fusion of IT and CT in the age of digital transformation, and heading toward ICT enterprises. Large global ICT enterprises, such as Apple, Google, Microsoft, Amazon, Verizon, and AT＆T, have been contributing to the performance improvement of IT service and CT equipment.

Application of 3D Static Modelling in Reservoir Characterization:A Case Study from the Qishn Formation in Sharyoof Oil Field,Masila Basin,Yemen

Three-dimensional(3 D)static modelling techniques are applied to the characterization of the Qishn Formation(Fm.)in the Sharyoof oil field locating within the Masila basin,southeastern Yemen.The present study was initiated by the seismic structural interpretation,followed by building a 3 D structural framework,in addition to analysing well log data and from these,3 D facies and petrophysical models are constructed.In the Sharyoof oil field,the Qishn Fm.exhibits depth values within the range of 400-780 m below sea level,with a general increase towards the SSE.A set of high dip angle normal faults with a general ENE-WSW trend dissect the rocks.The strata are also folded as a main anticline with an axis that is parallel to the fault trend,formed as a result of basement uplift.According to the facies models,the Qishn Fm.comprises 43.83% limestone,21.53% shale,21.26% sandstone,13.21% siltstone and 0.17% dolomite.The Qishn Carbonates Member has low porosity values making it a potential seal for the underlying reservoirs whereas the Upper Qishn Clastics SI A and C have good reservoir quality and SIB has fair reservoir quality.The Upper Qishn Clastics S2 and S3 also have fair reservoir quality,while the Lower Qishn Clastics zone has good reservoir quality.The water saturation decreases towards the west and east and increases towards north and south.The total original oil in-place(OOIP)of the Upper Qishn clastics is 106 million STB within the SI A,SIC and S2 zones.Drilling of development wells is recommended in the eastern study area,where good trapping configuration is exhibited in addition to the presence of a potential seal(Upper Qishn Carbonates Member)and reservoir(Qishn Clastics Member)with high porosity and low water saturation.

A Business Operation Stability by Improving the Speed ofRestoration on Software Application Service

Software application is still a heavy dependence for most of the business operation today.Whenever software application encounters error that causes downtime in the production environment,the root cause of the error can be either within the software application layer or any other factor outside the software application layer.To accurately identify the root cause is difficult whenever more than one log file is required for the root cause analysis activity.Due to such complexity,it leads to the entire duration on the root cause analysis activity became prolong.This will increase the total time taken on restoring the software application service back to the users.In order to identify the root cause of software application error in a more accurate manner,and shorten the duration of root cause analysis activity conducting on software application error,a Prescriptive Analytical Logic Model incorporates with Analytic Hierarchy Process(AHP)is proposed.The proposed Logic Model along with the algorithm will contribute a new knowledge in the area of log file analysis to shorten the total time spent on root cause analysis activity.

A preliminary study on exploratory search behavior of undergraduate students in China

Purpose: This study attempts to investigate how a user's search behavior changes in the exploratory search process in order to understand the characteristics of the user's search behavior and build a behavioral model.Design/methodology/approach: Forty-two matriculated full-time senior college students with a female-to-male ratio of 1 to 1 who majored in medical science in Jilin University participated in our experiment. The task of the experiment was to search for information about 'the influence of environmental pollution on daily life' in order to write a report about this topic. The research methods include concept map, query log analysis and questionnaire survey.Findings: The results indicate that exploratory search can significantly change the knowledge structure of searchers. As searchers were moving through different stages of the exploratory search process, they experienced cognitive changes, and their search behaviors were characterized by quick browsing, careful browsing and focused searching.Research limitations: The study used only one search topic, and there is no comparision or control group. Although we took search habits, personal thinking habits, personality characteristics and professional background into account, a more detailed study to analyze the effects of these factors on exploratory search behavior is needed in our further research.Practical implications: This study can serve as a reference for other researchers engaged in the same effort to construct the supporting system of exploratory search.Originality/value: Three methods are used to investigate the behavior characteristics during exploratory search.

Detecting compromised email accounts via login behavior characterization

The illegal use of compromised email accounts by adversaries can have severe consequences for enterprises and society.Detecting compromised email accounts is more challenging than in the social network field,where email accounts have only a few interaction events(sending and receiving).To address the issue of insufficient features,we propose a novel approach to detecting compromised accounts by combining time zone differences and alternate logins to identify abnormal behavior.Based on this approach,we propose a compromised email account detection framework that relies on widely available and less sensitive login logs and does not require labels.Our framework characterizes login behaviors to identify logins that do not belong to the account owner and outputs a list of account-subnet pairs ranked by their likelihood of having abnormal login relationships.This approach reduces the number of account-subnet pairs that need to be investigated and provides a reference for investigation priority.Our evaluation demonstrates that our method can detect most email accounts that have been accessed by disclosed malicious IP addresses and outperforms similar research.Additionally,our framework has the capability to uncover undisclosed malicious IP addresses.

An unsupervised anomaly detection framework for detecting anomalies in real time through network system’s log files analysis

Nowadays,in almost every computer system,log files are used to keep records of occurring events.Those log files are then used for analyzing and debugging system failures.Due to this important utility,researchers have worked on finding fast and efficient ways to detect anomalies in a computer system by analyzing its log records.Research in log-based anomaly detection can be divided into two main categories:batch log-based anomaly detection and streaming log-based anomaly detection.Batch log-based anomaly detection is computationally heavy and does not allow us to instantaneously detect anomalies.On the other hand,streaming anomaly detection allows for immediate alert.However,current streaming approaches are mainly supervised.In this work,we propose a fully unsupervised framework which can detect anomalies in real time.We test our framework on hdfs log files and successfully detect anomalies with an F-1 score of 83%.

Cognition:Accurate and Consistent Linear Log Parsing Using Template Correction

Logs contain runtime information for both systems and users.As many of them use natural language,a typical log-based analysis needs to parse logs into the structured format first.Existing parsing approaches often take two steps.The first step is to find similar words(tokens)or sentences.Second,parsers extract log templates by replacing different tokens with variable placeholders.However,we observe that most parsers concentrate on precisely grouping similar tokens or logs.But they do not have a well-designed template extraction process,which leads to inconsistent accuracy on particular datasets.The root cause is the ambiguous definition of variable placeholders and similar templates.The consequences include abuse of variable placeholders,incorrectly divided templates,and an excessive number of templates over time.In this paper,we propose our online log parsing approach Cognition.It redefines variable placeholders via a strict lower bound to avoid ambiguity first.Then,it applies our template correction technique to merge and absorb similar templates.It eliminates the interference of commonly used parameters and thus isolates template quantity.Evaluation through 16 public datasets shows that Cognition has better accuracy and consistency than the state-of-the-art approaches.It also saves up to 52.1%of time cost on average than the others.

Roundtable： Research Opportunities and Challenges for Large-Scale Software Systems

For this special section on software systems, six research leaders in software systems, as guest editors tor this special section, discuss important issues that will shape this field＇s future research directions. The essays included in this roundtable article cover research opportunities and challenges for large-scale software systems such as querying organization- wide software behaviors （Xusheng Xiao）, logging and log analysis （Jian-Ouang Lou）, engineering reliable cloud distributed systems （Shan Lu）, usage data （David C. Shepherd）, clone detection and management （Xin Peng）, and code search and beyond （Qian-Xiang Wang）. - Tao Xie, Leading Editor of Software Systems.

PLQ:An Efficient Approach to Processing Pattern-Based Log Queries

As software systems grow more and more complex,extensive techniques have been proposed to analyze the log data to obtain the insight of the system status.However,during log data analysis,tedious manual efforts are paid to search interesting or informative log patterns from a huge volume of log data,named pattern-based queries.Although existing log management tools and DMBS systems can also support pattern-based queries,they suffer from a low efficiency.To deal with this problem,we propose a novel approach,named PLQ(Pattern-based Log Query).First,PLQ organizes logs into disjoint chunks and builds chunk-wise bitmap indexes for log types and attribute values.Then,based on bitmap indexes,PLQ finds candidate logs with a set of efficient bit-wise operations.Finally,PLQ fetches candidate logs and validates them according to the queried pattern.Extensive experiments are conducted on real-life datasets.According to experimental results,compared with existing log management systems,PLQ is more efficient in querying log patterns and has a higher pruning rate for filtering irrelevant logs.Moreover,in PLQ,since the ratio of the index size to the data size does not exceed 2.5%for log datasets of different sizes,PLQ has a high scalability.

A Study of Detecting Student Engagement Modes in an Online Learning Platform

In this paper,we used the platform log data to extract three features(proportion of passive video time,proportion of active video time,and proportion of assignment time)aligning with different learning activities in the Interactive-Constructive-Active-Passive(ICAP)framework,and applied hierarchical clustering to detect student engagement modes.A total of 840 learning rounds were clustered into four categories of engagement:passive(n=80),active(n=366),constructive(n=75)and resting(n=319).The results showed that there were differences in the performance of the four engagement modes,and three types of learning status were identified based on the sequences of student engagement modes:difficult,balanced and easy.This study indicated that based on the ICAP framework,the online learning platform log data could be used to automatically detect different engagement modes of students,which could provide useful references for online learning analysis and personalized learning.