基于微博API的分布式抓取技术

A Distributed Data-Crawling Technology for Microblog API

随着微博用户的迅猛增长,越来越多的人希望从用户的行为和微博内容中挖掘有趣的模式。针对如何对微博数据进行有效合理的采集,提出了基于微博API的分布式抓取技术,通过模拟微博登录自动授权,合理控制API的调用频次,结合任务分配控制器高效地获取微博数据。该分布式抓取技术还结合时间触发和内存数据库技术实现重复控制,避免了数据的重复爬取和重复存储,提高了系统的性能。本分布式抓取技术具有可扩展性高、任务分配明确、效率高、多种爬取策略适应不同的爬取需求等特点。新浪微博数据爬取实例验证了该技术的可行性。

As more and more users begin to use microblog, people eagerly want to dig interesting patterns from the microblog data. How to efficiently collect data from the service provider is one of the main challenges. To address this issue, a distributed crawling solution based on microblog API was present. The distributed crawling solution simulates microblog login, automatically gets authorized, and control the invoked frequency of the API with a task controller. A time trigger method with memory database was also proposed to avoid extra trivial data duplication and improve efficiency of the system. In the distributed framework, the crawling tasks can be assigned to distributed clients independently, which ensures the high scalability and flexibility of the crawling procedure. The feasibility of the crawler technology according to Sina microblog instance was verified.