藏文词向量相似度和相关性评测集构建

Construction of Tibetan Words Embedding Similarity and Relevance Evaluation Set

词向量评测是词向量研究的基础,包括内部评测(intrinsic evaluation)和外部评测(extrinsic evaluations)。外部评测是将得到的词向量应用到具体某个任务中进行评测,是词向量研究的目标。内部评测是通过建立词之间的语义相似度或相关性能力的评测集,评价词向量模型的性能,是一种常用的词向量评测方式。该文通过分析英文、汉文词向量评测集构建方法,结合藏文的特点,研究藏文词向量评测集构建方法,构建了用于评价藏文词向量相似度和相关性的评测集TWordSim215和TWordRel215,并分析其有效性。

Evaluation of words embedding as an essential issue in the research can be performed by intrinsic evaluation or extrinsic evaluation.The intrinsic evaluation,as a basic solution,usually demands an evaluation set describing the similarity or relevance among words.After examing the construction methods of words embedding evaluation sets of English and Chinese,this paper investigate the construction of Tibetan words embedding evaluation set according to the characteristic of Tibetan.The evaluation sets WordSim215 and TWordRel215 are constructed and analyzed for their effectiveness of evaluating Tibetan words embedding similarity and relevance.