基于深度学习的中文分词方法研究

Research on Chinese Word Segmentation Based on Deep Learning

作为中文自然语言处理中的基础任务中文分词,其分词的好坏直接影响之后的自然语言处理任务。当前中文分词大部分都是采用基于机器学习的方法,但是其需要人工构建大量特征。针对上述问题,论文提出一种基于深度学习的新分词模型,该模型基于BLSTM(双向长短期神经网络),CNN(卷积神经网络)和CRF(条件随机场),充分利用了BLSTM可以利用长距离信息和CNN提取局部信息的优点。并设计了实验,在三个数据集上验证论文提出的模型在中文分词上的正确性和优越性。

As a basic task of Chinese word segmentation(CWS)in Chinese natural language processing,its segmentation directly determines the future of other natural language processing tasks. Most of the current Chinese word segmentation uses machine learning-based methods,but it requires manual construction of a large number of features. Aiming at the above problems,this paper proposes a new segmentation model based on deep learning. The model is based on BLSTM,CNN and CRF,making full use of BLSTM to make use of long distance information and the advantages of CNN extracting local information. Experiments are designed and the correctness and superiority of the proposed model in Chinese word segmentation are verified on three data sets.