一种基于LSTM的端到端多任务老挝语分词方法

An End-to-end Multi Task Method for Laotian Word Segmentation via LSTM

老挝语是一种无空格切分的字母语言,在进行自然语言处理工作时需要首先进行分词处理。现有分词算法主要为首先使用规则进行音节切分,然后根据音节切分结果进行老挝语分词,存在错误传递等问题。该文提出一种基于神经网络的端到端老挝语分词方法,基于多任务联合学习思想,将老挝语音节切分与分词工作进行结合,实现了基于双向长短时记忆循环神经网络(BiLSTM)的端到端老挝语分词模型。实验表明,端到端的老挝语分词模型准确率达到89.02%,较以往分词模型有所提升。

Laotian is a non-space separated alphabetic language. The existing segmentation algorithms for Laotian mainly use rules to segment syllables first, and then segment words according to the results of syllable segmentation. This paper proposes an end-to-end Laotian word segmentation method based on neural networks. With multi-task joint learning, the Lao syllable segmentation and word segmentation are jointly processed via BiLSTM. Experiments show that the precision of the proposed method reaches 89.02%, out-performing previous word segmentation models.