数据挖掘中Web文档转换算法的设计与实现

DESIGN AND IMPLEMENTATION OF WEB DOCUMENTS CONVERSION ALGORITHM IN DATA MINING

Web文本挖掘是数据挖掘技术在网络信息处理中的一个重要应用,如何将web文档转换成数据挖掘所要求的格式,即web文档预处理是一项很重要的研究课题.本文的方法是:从Internet网上下载了大量的网页文件,将网页文件转换成文本文件,然后通过算法对这些文本文件中的数据进行词频统计,删除非用词,去掉高频词,对单词进行词根处理,建立用词词表,从而抽取用词,按字母排序生成词频索引,和字典文件进行对照,获取单词的ID,最后生成Reuters-21578的Database数据格式.这样就将web文档数据转换成标准的数据集,以便为数据挖掘中分类、聚类作好准备.

Web text information mining is one of the important applications of applying data mining technologies into informa- tion analysis and processing, how to transform web documents into data mining to the required format, i.e. web document pre- processing becomes a significant research task. In this paper the method is ： from Internet to download a large number of web- page files, webpage files are converted into a text files, and then through the algorithm to word frequency statistics the data of the text files, delete non-using words, remove high frequency words, process etyma of substantive words, extract stems, elimi- nate redundant words and establish word lis4 thus extraction word list, alphabetical index to generate word frequency index, and the dictionary file comparison, get the word ID, the last generation of Reuters-21578 Database data format. This web docu ment data converted into standard data sets for classification and clustering to prepare in data mining.