结合日志与深度学习的网络软件异常检测算法

Network Software Anomaly Detection Algorithm Based on Log and Deep Learning

为了快速的检测出网络软件异常,提出基于日志与深度学习的网络软件异常检测算法。首先通过CNN卷积核和特征向量对文本向量的权重进行提取,将池化向量代入RNN中,选择Tf-Idf模型矩阵降低文本数据不均衡造成的影响。然后对非结构化的日志使用SPELL方法进行解析,转化成结构化日志,通过分隔符将转化为单序列的日志消息与消息列表中的公共常量进行匹配,解析出剩余变量,完成日志的解析工作。训练过程中,选择Adam优化算法使目标函数取得最小值,同时采用学习率衰减算法对参数的更新速率进行控制,通过反向传播对权重参数进行实时更新。最后,分别从执行路径异常检测、参数异常检测和在线更新模型三方面进行评估。实验结果表明,上述方法不仅能够准确的判断出异常,而且综合性能指标和召回率也较高,具有广泛的适用性。

In order to quickly detect network software anomalies, a network software anomaly detection algorithm based on log and deep learning is proposed. Firstly, the weight of the text vector was extracted through the convolution kernel and eigenvector of CNN,the pooled vector was substituted into the RNN,and the TF-IDF model matrix was selected to reduce the impact caused by the imbalance of text data. Then, the unstructured log was parsed using the SPELL method and transformed into a structured log. The log messages transformed into a single sequence were matched with the public constants in the message list through the separator, and the remaining variables were parsed to complete the log parsing. In the training process, the Adam optimization algorithm was selected to minimize the objective function. At the same time, the learning rate attenuation algorithm was used to control the update rate of parameters, and the weight parameters were updated in real-time through backpropagation. Finally, the execution path anomaly detection, parameter anomaly detection and online update model were evaluated respectively. The experimental results show that this method can not only accurately judge the abnormality, but also has high comprehensive performance index and recall rate, and has wide applicability.