基于Spark Streaming的僵尸主机检测算法

Detection bots using Spark Streaming

随着宽带互联网的广泛应用,产生了同时针对互联网服务提供商和用户的新型威胁,僵尸网络。僵尸网络通过多类传播和感染程序,构建一个可一对多控制的网络,操控大量僵尸主机发起DDo S攻击、发送垃圾邮件、偷窃敏感数据和钓鱼等恶意行为。基于一种分布式实时处理框架,提出一种分布式的僵尸主机检测算法。该算法能够充分利用网络流量的统计数据IPFIX,在无须深度包解析的情况下,能够识别僵尸主机行为。同时,使用该算法实现了IPFIXScanner原型系统。系统的鲁棒性和可扩展性是设计该系统的核心原则。实验表明,IPFIXScanner原型在使用指定僵尸家族样本训练的情况下,对于特定类的僵尸主机能够获得较高的检出率和较低的误报率。在核心交换机上的测试结果表明,IPFIXScanner能够进行分布式的实时检测,加速比接近线性,验证了Spark Streaming引擎在分布式流处理方面的优势,以及用于僵尸主机检测方面的可行性。

The wide use of broadband Internet connections has given rise to a new threat against Internet service providers and end users as well. Botnets are vast networks of compromised hosts under the control of single masters who possess the ability to launch large-scale malicious activities such as spamming,DDo S attacks,identity theft,and phishing with privacy-violating spyware and other forms of malicious software. This paper＇s goal is to introduce a novel distributed algorithm for malicious（ potential bots） activity recognition based on network traffic statistics generated by IPFIX,where IPFIX related is correlated as the host IPFIX graph structure,and a feature extraction method based on the spark streaming is leveraged for exacting implicit characteristics. Furthermore,in order to verify the validity of the algorithm,this paper established an IPFIXScanner detection prototype system. Scalability and robustness were the main principles during the design of the system architecture. The experimental results show that the IPFIXScanner is able to detect botnet participant computers（ bots） with the help of novel features originating from various local networks,while the algorithms provide utmost anonymity to network operators,and has approximate linear speedup. It proves the feasibility of applying spark streaming engine to distributed bots detection.