基于字频分布的中文网页编码识别算法

Chinese Webpage Encoding Identification Algorithm Based on Word Frequency Distribution

编码识别是网页内容过滤的必要前提,多种中文编码共存给中文网页的内容过滤带来不便。针对上述问题,提出一种基于字频分布的中文网页编码识别算法。根据汉字的使用频率,选取使用频度较高的字符构成高频字符编码表,以高频字符编码作为关键字,使用改进的模式匹配算法查找待识别网页,并统计匹配次数。将编码的匹配结果作为分析的依据,最终判定待识别网页的真实码制。实验结果证明,与Unigram算法相比,该算法对目前通用的中文编码识别率较高,适合对未知编码的中文网页进行快速编码识别。

Web coding identification is the premise of webpage content filtering,and coexistence of a variety of Chinese encoding makes Chinese webpage coded identification inconvenient. This paper presents a Chinese Web encoding identification algorithm———FKI （ Frequency Keyword Identification ） which is based on the frequency of Chinese character used. FKI selects the frequency of high character to construct high frequency character encoding tables, according to the frequency of the use of Chinese characters. Using high frequency character encoding as a keyword,FKI algorithm scans the Webpage by improved pattern matching algorithm, statistical matching number, and determines the real code of webpage based on the matching result. Experimental results show that, compared with the Unigram algorithm,this algorithm has a higher recognition rate. FKI algorithm is suitable for Chinese webpage which is unknown code to identify code quickly and accurately.