An Enhanced Automated Signature Generation Algorithm for Polymorphic Malware Detection

Polymorphic malware is a secure menace for application of computer network systems because hacker can evade detection and launch stealthy attacks. In this paper, a novel enhanced automated signature generation （EASG） algorithm to detect polymorphic malware is proposed. The EASG algorithm is composed of enhanced-expectation maximum algorithm and enhanced K-means clustering algorithm. In EASG algorithm, the fixed threshold value is replaced by the decision threshold of interval area. The false positive ratio can be controlled at low level, and the iterative operations and the execution time are effectively reduced. Moreover, the centroid updating is realized by application of similarity metric of Mahalanobis distance and incremental learning. Different malware group families are partitioned by the centroid updating.