Large-Scale KPI Anomaly Detection Based on Ensemble Learning and Clustering

Anomaly detection using KPI(Key Performance Indicator)is critical for Internet-based services to maintain high service availability.However,given the velocity,volume,and diversified nature of monitoring data,it is difficult to obtain enough labelled data to build an accurate anomaly detection model for using supervised machine leaning methods.In this paper,we propose an automatic and generic transfer learning strategy:Detecting anomalies on a new KPI by using pretrained model on existing selected labelled KPI.Our approach,called KADT(KPI Anomaly Detection based on Transfer Learning),integrates KPI clustering and model pretrained techniques.KPI clustering is used to obtain the similarity of different KPI data's distribution,and applied transfer knowledge from source dataset to the target dataset by model pretrained technique.In our evaluation using real-world KPIs from large Internet-based services,the clustering algorithm used to detect various KPI curve pattern achieve the best classification effect and accuracy More importantly,further evaluation on 30 KPIs shows that KADT can significantly reduce the time overhead of the model training with little loss of accuracy.