一种基于成词率和谱聚类的电力文本领域词发现方法

A power text domain word discovery method based on word formation rate and spectral clustering

考虑到当前电力行业仍缺少有效的领域词发现方法,以电力行业科技项目文本为原始语料库,将基于互信息与左右熵的统计特征与传统语言构词规则特征相融合,提出了电力文本成词率的概念。所提方法首先利用成词率对电力文本进行无监督筛选得到初始候选词集,然后对候选词集进行文本切片算法和常用词过滤操作,最后进行词嵌入和谱聚类得到最终所需的电力文本领域词。实验结果表明,所提出的方法准确有效,为电力文本的领域词发现提供了一种新方法。

Considering that the current power industry still lacks effective domain word discovery methods,this paper takes the power industry science and technology project text as the original corpus,combines the statistical features based on the mutual information,left entropy as well as right entropy with the features of traditional language word-formation rules,and proposes the new concept of power text word formation rate.The proposed method firstly uses the word formation rate to get the initial candidate word set by unsupervised filtering,and then performs the text slicing algorithm and common word filtering operation on the candidate word set,and finally performs the word embedding and spectral clustering algorithms to get the final power text-domain words.Experimental results show that the method proposed in this paper is accurate and effective,and provides a new method for power text domain word discovery.