基于Hadoop的可视化Deep Web采集平台设计

Design of a visual Deep Web crawler platform based on Hadoop

随着信息技术的发展,互联网信息资源变得越来越丰富,大数据技术的发展使得我们能够从互联网复杂的信息数据中获得相应的知识。这其中最基本的技术就是大数据采集技术,它使我们能够将互联网数据快速采集下来并结构化存储。设计的基于Hadoop的可视化Deep Web采集平台是一种简单易操作的高效深度采集平台,运用Webkit技术作为核心引擎实现可视化配置和深度采集功能,同时通过优化采集算法,调整Hadoop任务分配策略提升效率。实验结果表明,设计的数据采集平台获得了较好的效果。

With the development of IT technology, internet information resources become much richer. We can obtain relevant knowledge from complicated internet information thanks to the rapid development of big data technology. The most essential part is the big data crawler technology which can crawl and save Internet data structurally. In this paper, we present and develop an efficient Deep Web information crawler based on Hadoop. This crawler employs the Webkit as the core engine which can implement the visual configuration and the deep data collection. To improve the efficiency, the data collection algorithm is also optimized by adjusting the strategy of task distribution in Hadoop. Experimental results demonstrate that the developed data collection platform can obtain better results.