一种基于动态网页解析的微博数据抓取方法

Date Crawler for Sina Microblog Based on Dynamic Webpage Date Interpreting

微博是一种新型信息传播媒介,产生的海量数据吸引研究人员关注并开展相关研究。微博数据获取是后续分析研究的基础和起点。以新浪微博平台为研究对象,提出了基于动态网页解析技术的微博数据多线程抓取方法。方法利用浏览器内核解析微博页面动态数据,通过模拟登陆,依据社交网站网页链接特点确定页面爬取策略,使用页面解析技术定制页面数据抽取模板,实现以用户为中心的微博数据获取。抓取结果表明,方法能对微博用户数据进行全面高效抓取,为后续数据分析和处理提供数据支持。

Microblogging is a new kind of information media.The mass data are generated to attracts the attention of the researchers to carry out related research.Micro-blog data acquisition is the basis and starting point for further research.This paper presents a multi-threaded crawler for Sina microblog platform based on dynamic webpage interpreting.The browser kernel is used to interpret the dynamic data of microblog webpage.Through simulated login,the page crawling strategy based on the characteristics of social networking site is determined,and the webpage parsing technology is used to custom templates of webpage to achieve user-centric microblog data acquisition.The test results show that the method can capture microblog data of user comprehensive and efficiently,provide data support for subsequent dta analysis and processing.