基于深度学习的相似语言短文本的语种识别方法

LANGUAGE IDENTIFICATION METHOD FOR SIMILAR LANGUAGE IN SHORT TEXT BASED ON DEEP LEARNING

在语种识别中,传统的N-Gram方法对文本长度依赖度高,因而无法有效地对短文本进行语种识别。现有的基于神经网络的模型无法同时考虑词本身信息和词间组合信息,从而降低了短文本语种识别的质量。针对以上问题,提出一种基于深度学习的字符级短文本语种识别方法。采用卷积神经网络从字符向量中获取词中字符组合信息;通过长短期记忆网络获取词与词之间的特征信息;使用全连接网络实现相似语言的语种识别。在维吾尔语、哈萨克语以及DSL2017数据集上的实验结果表明,该方法可以有效地提高相似语言短文本的识别精度。

In the language identification,the traditional N-Gram method has a high degree of dependence on the length of the text,so it cannot effectively identify the short text.Moreover,the existing models based on neural network cannot consider the information of the word itself and the combination of words at the same time,which reduces the quality of short text recognition.Aiming at the above problems,this paper proposes a character level short text language identification method based on deep learning.CNN was used to obtain the character combination information from the character vector.Then,LSTM was used to obtain the features between words.Finally,we used the full connection network to realize the language identification of similar languages.The experimental results on the corpus of Uyghur and Kazakh as well as DSL2017 show that this method can effectively improve the identification accuracy of short texts in similar languages.