基于大规模语料库的汉语词义相似度计算方法

Chinese Lexical Semantic Similarity Computing Based on Large-scale Corpus

词义相似度的计算是自然语言处理领域的关键问题之一,它在信息检索中的查询扩展、机器翻译中的模块识别,以及句法分析、词义消歧等任务中都发挥着重要的作用。该文研究了基于大规模语料库的汉语词义相似度计算方法,系统地比较分析了上下文特征权值的选择、向量相似度计算方法、基于窗口和基于依存关系的表征形式、新闻语体和网络语体的差异。实验结果表明,在网络语言语料上,基于窗口选取上下文特征,用互信息PMI来计算权值,采用cosine来计算相似度,取得了最好的词义相似度结果。

Automatic acquisition of similar words is one of the most crucial problems in natural language processing tasks, e.g. the query extension in information retrieval, pattern identification in machine translation, parser analysis and WSD. This paper focuses on Chinese semantic similarity computing based on large corpus, investigating the computation of context feature weight, the vector similarity measures, the window context vs. the dependency con text, and the newspaper corpus vs. web corpus. Our experiments show that, in the web corpus, using windowbased context combined with PMI weights function, the cosine measures gets the best semantic similarity results.