基于Nutch的Web论坛分块采集系统

Web Forum Information Acquisition System Based on Nutch

随着网络论坛用户的不断增长,论坛帖子数量急剧增加,论坛结构不断变化,研究适用于整个Web论坛信息采集解析的定向、自动、准确的采集方法是网络爬虫领域研究的难点。由论坛领域建模、论坛模板解析、论坛板块采集及论坛帖子采集四大模块构成的基于Nutch开源技术的采集系统,将分块解析理论应用于论坛模板解析中,较大提高了采集效率。39健康网数据爬取实例验证了该项技术的可行性。

With the increasing number of web forum users and the rapid increase in the number of forum posts, it is necessary to obtain a directional, automatic and accurate acquisition method which is suitable for the whole Web forum. This Web forum segmentation harvest system based on Nutch open source is integrated by forum domain modeling, forum template parsing, forum collection and forum post collection. The block analytical theory is applied to the parse of forum template, which greatly improves the efficiency of data collecting. The feasibility of the acquisition technology based on Nutch is verified by the data acquisition of 39 Health Websites.