基于联邦迁移学习的应用系统日志异常检测研究

Research on application system log anomaly detection based on federated transfer learning

迄今为止,基于日志的异常检测研究已经取得了很多进展,然而,在现实条件下仍旧存在两个挑战:(1)是日志数据通常以“数据孤岛”形式储存在不同的服务器上,单一公司或组织的日志数据中异常样本量不足,且异常模式较为固定,很难通过这些数据训练出一个准确率高的检测模型.为了解决这个问题,将不同来源的日志数据整合成更大的数据集可以提高模型训练的效果但可能会在数据传输过程中产生日志数据泄露问题;(2)是不同应用系统类型的日志数据通常在结构和语法上存在差异,简单地整合并用于训练模型效果不佳.基于以上原因,本文提出一种基于联邦迁移学习的日志异常检测模型训练框架LogFTL,该框架利用基于匹配平均的联邦学习算法,在保证客户端数据隐私安全的前提下于服务器聚合客户端的模型参数形成全局模型,再将全局模型分发给客户端并基于客户端的本地数据进行迁移学习,优化客户端本地模型针对自身常见异常行为的检测能力.经过实验表明,本文提出的LogFTL框架在联邦学习场景下效果超过了传统的日志异常检测方法,同时也证明了该框架中迁移学习的效果.

Significant progress has been made in the research of log anomaly detection.However,two challenges still exist in reality.Firstly,log data is often stored on different servers,creating"data islands",the number of abnormal samples in the log data of a single company or organization is insufficient and the abnormal patterns are relatively limited,it is a challenge to train a detection model with high accuracy through these data.Integrating log data from different sources can improve the model''s performance but may result in log data leakage during transmission;Secondly,the log data of different application system types varies in log structure and syntax,and simple integration for training models is ineffective.To address these issues,this paper proposes a log anomaly detection training framework called LogFTL based on federated transfer learning,which uses federated learning algorithm based on matching average.On the premise of ensuring the privacy and security of the client''s data,LogFTL aggregates the model parameters of the client on the server side to form a global model which is then distributed to the client side.Using the client''s local data,the LogFTL framework migrates and learns to optimize the client’s local model and the detection effect of local log data is improved.The experiment resluts show that the LogFTL framework proposed in this paper outperforms traditional log anomaly detection methods in federated learning scenarios,and demonstrate the transfer learning effectiveness of LogFTL.