融合主题模型及双语词向量的汉缅双语可比文档获取方法

Chinese-Burmese Comparable Document Acquisition Based on Topic Model and Bilingual Word Embedding

缅甸语属于资源稀缺型语言,汉缅双语可比文档是获取平行句对的重要数据资源。该文提出了一种融合主题模型及双语词向量的汉缅双语可比文档获取方法,将跨语言文档相似度计算转化为跨语言主题相似度计算问题。首先,使用单语LDA主题模型分别抽取汉语、缅甸语的主题,得到对应的主题分布表示;其次,将抽取到的汉缅主题词进行表征得到单语的主题词向量,利用汉缅双语词典将汉语、缅甸语单语主题词向量映射到共享的语义空间,得到汉缅双语主题词向量,最后通过计算汉语、缅甸语主题相似度获取汉缅双语可比文档。实验结果表明,该文提出的方法得到的F1值比基于双语词向量方法提升了5.6%。

To collect Chinese-Burmese comparable documents, this paper proposes a acquisition method based on topic model and bilingual word embedding, treating the cross-language document similarity issue as cross-language topic similarity measurement. First, we use the monolingual LDA topic model to extract the Chinese and Burmese topics, respectively, and get the corresponding topics distribution representation. Then, we calculate the topic words for Chinese and Burmese documents, and get the Chinese-Burmese bilingual topic word embedding by mapping the monolingual word embedding into a shared semantic space according the Chinese-Burmese bilingual dictionary. The similarity of Chinese and Burmese document is finally decided by both topic embedding and bilingual word embedding. The experimental results show that the F1 obtained by the proposed method is increased by 5.6% than the baseline using just the word embedding.