Twitter数据采集方案研究

Research of Twitter data collection

为了能够实时、高效地获取Twitter数据,在分析了传统采集方法的缺陷后,提出了基于Twitter List API和Lookup API的用户数据采集方案。该方案通过对用户进行分类,进而精确控制API的调用频率。经在超过26万Twitter用户和600万条消息的一系列实验证明,通过两套方案的结合可以实现Twitter用户数据高效实时的获取。

In order to achieve real-time and efficient access to the data of Twitter,two different methods based on Twitter List API and Lookup API were presented after analyzing the shortcomings of traditional collection methods.By classi-fying users,this method can precisely control the frequency of calling API.A series of experiments on over 260,000 users and over 6 million messages were carried out,and the results show that the combination of the two methods can be efficiently used to collect Twitter data in real-time.