摘要
针对日益突出的网页访问问题,设计了一种新型分布式Web Spider。该分布式Web Spider采用中央控制节点来协调各个web spider的行为,利用宽度优先搜索来获得高质量的网页,通过对DNS缓存来提高访问web server的速度,以增加并行线程数量的方式来增加网页下载速度,并能动态地加入web spider节点和子中央控制节点,具有很强的灵活性和扩张能力。实验结果表明该分布式WebSpider作为搜索引擎的前端能够快速有效地下载网页,具有较好的性能。
For the increasingly prominent web access problems, A New Distributed Web Spider （NDWS） was proposed. NDWS uses central control node to coordinate actions of all web spiders,employs Breadth First search to obtain high-quality web pages, caches DNS to improve speed of access to web server, increases number of concurrent threads to increase download speed of web pages. Meanwhile, NDWS also can dynamically add web spider node and sub-central-control-node so that NI）WS has strong flexibility and expansion capability. Experimental results show that as a front end of search engine, NDWS can quickly and efficiently download web pages, and has better performance.
出处
《计算机与数字工程》
2012年第9期63-65,共3页
Computer & Digital Engineering
