一种基于信息熵的Web页面主题信息抽取方法

Extracting topic information of Web page based on entropy

提出了一种剪枝信息熵增较大结点的信息抽取方法。通过对HTML文档解析来构造DOM树,根据配置过滤掉不需处理的相关内容并建立语义模型树,最后对熵增超过阈值的结点进行剪枝并输出抽取的主题信息页面,初步实验结果验证了用这种方法进行Web页面信息抽取的有效性。方法的数学模型简单可靠,基本不需要人工干预即可完成主题信息抽取。可应用于Web数据挖掘系统以及PDA等移动设备的信息获取方面。

This paper presents a method of information extraction by pruning the nodes of which information entropy production reach a certain extent.Firstly,a DOM tree is constructed by parsing HTML document.Then,the nodes which don＇t need to be dealt with are filtrated out,and a STU tree is created.Lastly,the nodes whose information entropy＇s increase overtops the threshold value are pruned,and the topic information of the Web pages is obtained.The primary experiment result proves the validity of the method using for extracting Web page＇s information.The mathematical model of the method is simple and credible,so it can work automatically without intervention of people.This method can be applied to Web data mining and information extraction for mobile device such as PDA etc.