基于Storm的实时计算框架的研究与应用

Research & Application of real-time compute framework based on Storm

互联网数据的增长,催生了一大批新的数据处理技术,Map Reduce,Hadoop及相关技术使得我们能够处理的数据量比以前要大得多,但这些技术的设计目的都不是为了实时计算。然而随着社交网络服务的流行,大规模的实时数据处理已经越来越成为一种业务需求。Twitter Storm的出现弥补了Hadoop在实时处理方面的不足。本文就Storm的组成、运行机制以及计算模型进行研究,并设计与实现了基于Storm的社交网络中热门话题的实时计算问题。

With the growth of Internet data, a large number of new data processing techniques born. MapReduce, Hadoop and other related technologies enable us to handle much more data. But none of these technologies are designed to real-time computing. However, with the popularity of social networking service, real-time big data processing has increasing become a business needs. Twitter Storm’s appearance makes up for the lack of Hadoop in real-time processing. In this paper, we will study Storm composition, operation mechanism and computational models. What is more, we will design and implement the issue of real-time computing of the hottest topic in social network based on Storm.