基于向量空间的网页内容相似度计算方法研究

Research on Webpage Content' Similarity Calculation Method Based on Vector Space Model

针对海量网页数据挖掘问题,提出基于向量空间的网页内容相似计算算法和软件系统框架。利用搜索引擎从海量网页中提取中文编码的网页URL,在此基础上提取网页的中文字符并分析提取出中文实词,建立向量空间模型计算网页内容间的相似度。该系统缩小了需要进行相似度计算的网页文档范围,节约大量时间和空间资源,为网络信息的分类、查询、智能化等奠定了良好的基础。

Aiming to data mining in great mass of Web pages,this paper puts forward Web page content＇ similarity calculation method based on vector space model and software system framework.This system extracts massive Web pages from search engines and distinguishes the URL pages coded in Chinese,then extracts this page out Chinese characters and selects out Chinese notional words,establishes vector space model to calculate the similarity between Web pages＇ contents.The system reduces the Web document range,saves a lot of time and space,and lays a good foundation for the classification,search and intellectualization for network information.