藏文文本相似度计算方法研究

Study on the Similarity Algorithm of Tibetan Text

随着藏文文献数量的增加,原创性的藏文文献保护需求也越来越迫切,一个准确有效的藏文文本相似度计算方法就显得十分重要。文章针对藏文文字特殊结构导致藏文相似度计算不能照搬中英文文本相似度计算方法的问题,尝试按照《藏文编码字符集》的标准对藏文字符以一定顺序编码后与词库中的近义词关联;然后将待测文本和对照文本向量化,再对向量化的文本进行关键词提取,并用各自获取的关键词修正向量;最后使用余弦相似度原理计算待测文本向量和对照文本向量的余弦值,以此表示两句话的相似度。针对关键词提取的有效性,文章分别研究了TF-IDF和TF-IWF两种方法在不同主题词语比例的语料库下召回关键词的情况,结果表明TF-IWF提取关键词时能降低语料库中不同主题词语比例对计算结果的影响。针对相似度计算结果的准确性,文章引入皮尔森相关系数进行结果准确度评价,基于TF-IWF的相似度计算方法的皮尔森相关系数为0.7108,表明该方法是一种适用于藏文文本相似度计算的有效方法。

With the increasing of the number of Tibetan literature,protection of original Tibetan literature is be⁃coming more and more urgent.Hence,an accurate and effective method to calculate the similarity of Tibetan text becomes very important.Due to the fact that the similarity calculation method of Tibetan text cannot be directly copied from that of Chinese and English text because of the special structure of Tibetan characters,in this paper we proposed a new similarity algorithm specially tailor to Tibetan text.According to the standard of Tibetan Cod⁃ed Character Set,Tibetan characters are firstly encoded in a certain sequence and associated them with syn⁃onyms in the thesaurus.Secondly,the tested text and the reference text are vectorized.Then,the vectorized text is extracted with keywords,and the obtained keywords are used to correct the vector.Finally,cosine similarity principle is used to calculate the cosine value of vectors of the tested and the reference text,which represents the similarity of the two sentences.To verify the effectiveness of keyword extraction,the retrieval of keywords using TF-IDF and TF-IWF methods from the corpus with different proportion of subject words are studied in this pa⁃per.It shows that TF-IWF can reduce the influence of different proportion of subject words in the corpus on the results of calculation.Pearson correlation coefficient,which is introduced in this paper to evaluate the accuracy of results,of similarity calculation method based on TF-IWF is found to be 0.7108,indicating this method is an effective method for similarity calculation of Tibetan texts.