词义层级上的专家系统问题相似度计算优化

Similarity Calculation Optimization of Expert System at Lexical Level

词向量(Word2Vec)是近些年来自然语言处理领域的重要算法,在近几年的人工智能发展中占有极其重要的地位。通过向量空间的形式对每一个词进行标志,进而在概率方面上对词进行表示。Word Mover Distance算法是Earth Mover Distance的一个特殊形式,用来计算一组向量之间最短距离。文章使用上述两个算法作为基底,对词向量进行相关的空间映射预处理操作,作为WMD(word mover distance)的输入参数,最终可以得到句子间相似度得分。实验表明,该方法使相似语句与不相关语句之间的距离差距更大,且在专家系统中相似问句之间的距离更加紧密,更能显著刻画句子之间的语义相似程度,有利于增加短文本匹配的准确度。

Word vector (Word2Vec) is an important algorithm in the field of natural language processing in recent years. Accordingly, it has a very important position in the development of artificial intelligence. It can express each word in the form of vector space, and it can express the word in terms of probability. The Word Mover Distance algorithm is a special form of the Earth Mover Distance used to calculate the shortest distance between a set of vectors. Based on the above two algorithms, the article uses the word vector for the relevant spatial mapping preprocessing operation to get word mover distance (WMD) input parameters, and ultimately the similarity score of two words could be obtained. Experiments show that the method can make the distance between the similar statements and the distance between the irrelevant statements greater, and the distance between the similar question sentences in the expert system closer. Thus, the semantics between sentences similarity is more clearly described, so as to increase the accuracy of short text matching.