用有向图法解决网页爬行中循环链接问题

Solve the cycle links problem in Internet crawlingby directed graph

提出网页构成的有向回路问题,描述了由网页构成有向图的形式定义,并给出了用有向图法发现网页构成的有向回路算法.所给定的算法能使网页爬行器避免掉入由已爬行过的网页构成的有向回路陷阱.

The present paper deals with the technique how to solve the problem of cycle links in internet crawling by directed graph. First, the problem is proposed. Then, the formal definition of cycle links in internet crawling is described. Finally, the algorithm to solve the problem by directed graph is given. The key problem to a crawler is how to find directed loops effectively in web pages crawled by the crawler. The algorithm described in this paper can make the crawler avoid dropping in the pitfall created by cycle links.