基于独立分量分析的隐蔽Web领域聚类

Hidden Web Domain Clustering Based on Independent Component Analysis

针对隐蔽Web主题领域自动识别问题,提出一种基于独立分量分析(ICA)的聚类算法。对查询页面进行页面文本抽取和预处理,利用TF-IDF公式计算权重并选择前N个权重最大的特征词构造文档矩阵,在使用潜在语义索引(LSI)进行特征重构的基础上通过ICA分解获得类别信息。利用LSI的词共现分析和文本降噪能力提高聚类准确率。实验表明聚类平均准确率达到90%以上。

Aiming at organizing hidden Web databases according to their topic domains, this paper proposes an Independent Component Analysis（ICA） based algorithm for hidden Web domain clustering. Text is extracted from search interface pages as common Web pages, and TF-IDF formula is applied to weight terms. After selecting the top N-highest weight terms to construct VSM, the algorithm performs a singular value decomposition to implement features reconstruction. It applies ICA decomposition to obtain the cluster information. The main idea is utilizing the co-occurrence analysis and noise eliminating ability of Latent Semantic Index（LSI） to improve cluster performance. Experiment shows that the average precision is higher than 90 percent.