基于CRF模型的维吾尔语分词研究

Uygur word segmentation based on conditional random fields model

条件随机场能够很好地处理序列标注问题.引入条件随机场进行维吾尔语分词方法研究,主要包括制定词性和分词单独标注与一体化标注集并建成语料库;设计不同特征模板进行训练测试,反复比较实验结果,总结优化以获取最佳的特征模板.在设计特征模板时充分结合维吾尔语语言形态特征,采用了对称特征组合非对称特征的设计方法,并将获得的最佳分词模板应用到分步预测词性和分词实验中.相比单独分词标注,分词时加入词性特征列进行分词与词性一体化标注能展现更优的分词性能.

CRF(Conditional random fields),a word segmentation algorithm is introduced to handle sequence labeling problems.The main tasks include the formulation of the corresponding tag sets,part of speech tagging and integrated tagging.Furthermore,the design of feature templates fully combined with the morphological features of Uyghur language and asymmetric features are focused on,which are applied to building corresponding segmentation models.The experiments are carried out repeatedly using different templates in order to obtain the best one.Compared with the separate word segmentation,the participle feature column used for word segmentation performs better.