语言知识驱动的词嵌入向量的可解释性研究

A Study of Knowledge Motivated Explainalbe Word Embedding Vector

神经网络语言模型应用广泛但可解释性较弱,其可解释性的一个重要而直接的方面表现为词嵌入向量的维度取值和语法语义等语言特征的关联状况。先前的可解释性工作集中于对语料库训得的词向量进行知识注入,以及基于训练和任务的算法性能分析,对词嵌入向量和语言特征之间的关联缺乏直接的验证和探讨。该文应用基于语言知识库的伪语料法,通过控制注入语义特征,并对得到的词嵌入向量进行分析后取得了一些存在性的基础性结论:语义特征可以通过控制注入到词嵌入向量中;注入语义特征的词嵌入向量表现出很强的语义合成性,即上层概念可以由下层概念表示;语义特征的注入在词嵌入向量的所有维度上都有体现。

Neural network language models have many applications without much interpretations.An important and direct aspect of its interpretability is the association between word embedding vectors and linguistic features.The previous work of interpretability focuses on the knowledge injection to corpus-based word embedding and the theoretical analysis of training models,without direct verification and discussion on the correlation between word embedding vectors and linguistic features.In this paper,the pseudo-corpus derived from knowledge bases is applied.Some preliminary findings include:1)it is feasible to inject semantic features into the word embedding vectors under control;2)the compositionality of the word embedding vectors,i.e.the upper concept can be represented by the lower concepts,is observed with injected linguistic features;3)the injection of semantic features is reflected in all dimensions of word embedding vectors.