Web结构挖掘中基于熵的链接分析法

Entropy-based link analysis algorithm for web structure mining

在Web结构挖掘中,传统的HITS(hyperlink induced topics search)算法被广泛应用来寻找搜索引擎返回页面中的Autho-rity页面和Hub页面。但是在网站中除了有价值的页面内容外,还有很多与页面内容无关的链接,如广告、链接导航等。由于这些链接的存在,应用HITS算法时就会导致某些广告网页或无关网页获得较高的Authority值和Hub值。为了解决这个问题,在原有HITS算法的基础上,引入了香农信息熵的概念,提出了基于熵的网页链接分析方法来挖掘网页结构。该算法的核心思想是用信息熵来表示链接文本所隐含的知识。

In Web structure mining, hyperlink induced topics search （HITS） algorithm has been widely employed to analyze authorities and hubs of pages returned by search engine. However, except for useful information, most of content sites contain some irrelevant hyperlinks, such as advertisements and navigation panels. And because of these extra hyperlinks, HITS is found insufficient in analyzing advertisement or irrelevant pages, which would result in high authority values or hub values for these pages. In order to solve this problem, Shannon information entropy is introduced to HITS algorithm, thus the entropy-based link analysis algorithm is presented to mine Web informative structures. The key idea of this algorithm is to utilize shannon information entropy to represent the knowledge hided in link texts.