利用N-gram和语义分析的维吾尔语文本相似性检测方法

Uyghur text similarity detection method using N-gram and semantic analysis

为了实现维吾尔语文本的相似性检测,提出一种基于N-gram和语义分析的相似性检测方法。根据维吾尔语单词特征,采用了N-gram统计模型来获得词语,并根据词语在文本中的出现频率来构建词语-文本关系矩阵,并作为文本模型。采用了潜在语义分析(LSA)来获得词语及其文本之间的隐藏关联,以此解决维吾尔语词义模糊的问题,并获得准确的相似度。在包含重组和同义词替换的剽窃文本集上进行实验,结果表明该方法能够准确有效地检测出相似性。

In order to detect similarities between Uighur texts, this paper proposed a similarity detection method based on N-gram and semantic analysis. Firstly, it used N-gram statistical model to obtain the words based on Uyghur word features, and constructed the word-text relation matrix according to the appearance frequency of the words in the text. Then, it adopted an LSA to obtain the hidden association between the words and their texts, so as to solve the problem of vague semantic meaning in Uyghur language and obtain exact similarity. Experiments on plagiarized text sets containing reorganization and synonym replacement show that this method can detect the similarity accurately and effectively.