Web数据的深度定向采集

Deep directional collection of Web data

通过模拟人类访问网页的浏览行为,提取定向爬行子页面集限定爬虫的爬行方向;引入页面继承关系,并通过爬行条目的属性继承实现跨页面复合对象的数据关联关系;设计实现了支持深度定向采集的通用爬行流程。面向天涯热帖的舆情采集实验结果表明:该方法可以在整体处理流程不变的前提下,实现复杂对象的数据采集,并具有较高的采集效率。

Based on the Web surf behaviors of human beings,crawling directions are restricted by extracted crawling sub-pages,and the associated relationships of cross-page compound object are realized through the properties′ inheritance between crawl datum.Then,the generalized crawl process with deep directional collection support is designed and implemented.Experimental results about the hot posts of the Tianya site show that this method can achieve data collection of complicated objects without changing the main procedure,and has high collection efficiency.