面向音素序列的黏着语词干提取研究

Phoneme Sequence Based Stemming of Agglutinative Language

针对当前的黏着语词干提取任务难以处理具有上下文信息的句子级语料的问题,本文将维吾尔语作为研究对象,提出了一种句子上下文和字符特征相融合的,由BiLSTM、注意力机制(Attention)和CRF构成的词干提取模型.首先以句子级别的字符特征向量为输入,使用BiLSTM模型获取正向和反向的上下文序列特征,并在此模型上加入注意力机制进行权重学习,通过提取全局特征信息来捕获词干和词缀边界;最后添加CRF使其从序列特征中学习更多信息,从而更有效地描述上下文信息.为验证上述模型的有效性,将本文模型在两种不同的数据集上进行了实验,并且将本文模型跟传统模型进行了对比.实验结果表明,本文模型对于句子级语料的效果更好,可以更有效地提取词干.此外,本文提出的模型优于其他传统模型,能全面考虑数据特征,具有一定的优越性.

For the problem that the current agglutinative language stemming task is difficult to deal with sentence-level corpus with context information,this paper takes Uyghur language as the research object,and proposes a stemming model composed of BiLSTM,Attention and CRF,which integrates sentence context and character features.First,the sentence-level character feature embedding is used as input,and the BiLSTM model is used to obtain the forward and backward context sequence features,and the Attention Mechanism is added to this model for weight learning,and capture stem and affix boundaries by extracting global feature information;Finally,the CRF is added to make it learn more information from the sequence features,so that the context information can be described more effectively.In order to verify the effectiveness of the above model,the model in this paper is tested on two different datasets,and the model in this paper is compared with the traditional model.The experimental results show that the model in this paper is more effective for sentence-level corpus and can extract stems more effectively.In addition,the model proposed in this paper outperforms other traditional models,can fully consider the data characteristics,and has certain advantages.