基于改进的Prefixspan算法的中文文本新词提取方法研究

Research on the Method of Chinese Text New Word Discovery base on Improved Prefix Span Alporithm

该文尝试将序列模式挖掘算法Prefixspan应用于中文文本新词提取中,针对Prefixspan算法挖掘出的序列模式不连续、挖掘出的序列模式项相互间存在包含关系等问题,对算法进行改进,采用语义特征与统计相结合的方法,实现了从中文语料中有效提取新词。实验结果表明,该方法对于专业领域新词的识别具有较高的准确性。

The article attempts to apply the sequential pattern mining algorithm—Prefixspan to the extraction of Chinese text Neologisms.Aiming at the problem of sequential pattern discontinuity,the mining sequence patterns include each other and so on, the paper improved the prefixspan algorithm and combined semantic features with statistics to achieve effective discovery new words from Chinese text. The experimental results show that the method has high accuracy in the new word discovery.