基于编辑距离的无序词表的对齐和定位

Disorder word alignment and position based on Levenshtein Distance

语言调查采集到的数据存在相当程度的差异,需要进行二次加工。本文基于编辑距离算法实现从语言和方言词汇大数据中的词汇相似匹配及数据的对齐和定位。通过对达让语数据进行的3次实验发现,在做距离计算时,以词算而不是以词加括号内注释的整体去算的方式在保证抽取词汇召回率的基础上准确率会显著提升。实验结果表明,基于编辑距离的数据抽取方法是可行的,具有较好的检索效果。

The data of languages collected from field research have considerable differences,they need for secondary process.This paper implements the match and extraction of vocabulary similarity from the big data of language and dialect vocabulary based on levenshtein distance algorithm.Through the three experiments made by using the data of Darang,the research finds that the way of counting words rather than words and comments in brackets as a whole increases the precision rate dramatically on the basis of ensuring extraction vocabulary recall rate when the levenshtein distances are computed. The experimental results show that it is feasible to extract data based on levenshtein distance,the aboved method has better retrieval effect.