基于NS3的分布式消息系统Kafka的仿真实现

Simulate and Implement of Kafka distributed message system based on NS3

在数据已渗透到我们生活的各个领域的时代,人们对于数据的挖掘和使用愈发频繁。作为以消息为单位进行数据共享的分布式架构,分布式消息系统成为数据处理的核心技术。传统的分布式消息系统大多用于处理数据量小的关键性数据,然而在信息剧增的今天,人们对信息的关注领域在不断扩大,挖掘的信息量在不断增多,传统的消息处理架构已不能满足我们对数据的处理需求,一个高吞吐量,可实时消费的高性能分布式消息系统成为必需。Kafka即是一种处理海量数据的分布式消息系统[1]。本文总结了Kafka系统的特征和架构策略,对其进行抽象建模,通过网络仿真工具NS3[2],设计实际系统的场景部署,最后运行仿真系统,得出数据并分析,以帮助我们理解和评估Kafka分布式消息系统。

As data has penetrated into most fields of our life,the use of Data mining is more frequently. Distributed messaging system, which shares data on message basis,become the core technology gradually. With the extreme growing of information recently, the traditional system which focus on small account of data, cannot meet our demands for data processing. A high-throughput distributed messaging system with the function of real-time consumption become necessary. Kafka is such arequirement. This paper summarizes the characteristics and framework design of Kafka, then modelswith the network simulator tool NS3 and simulates the behavior of actual system, at last make conclusion through analysis the simu-lation result in order to realize and evaluate Kafka.