一种支持混合语言的并行查询纠错方法

Aparallel Query Correction Method for Mixed Language

中文信息检索系统中的查询语句包含中文字、拼音、英文等多种形式,而有些查询语句过长,不利于纠错处理。现有的查询纠错方法不能很好的解决中文检索系统中的混合语言与中文长查询的问题。为了解决上述两个问题,该文提出了一种支持混合语言的并行纠错方法。该方法通过对混合语言统一编码,建立统一编码语言模型和异构字符词典树,并根据语言特点制定相应的编辑规则对查询词语进行统一处理,其中,针对中文长查询,提出双向并行的纠错模型。为了并行处理查询语句,我们在字符词典树和语言模型的基础上提出了逆向字符词典树和逆向语言模型的概念。模型中使用的训练语料库是从用户查询日志、网页点击日志、网页链接信息等文件中提取的高质量文本。实验表明,与单向查询纠错相比,支持混合语言的并行纠错方法在准确率上提升了9%,召回率降低了3%,在速度上提升了40%左右。

Query in Chinese information retrieval system often contains Chinese,Chinese phonetic alphabet and English etc.Existing method can not solve the issue of mixed language and long Chinese query.In order to solve these problems,we propose a parallel query correction method for mixed language.The method establishes language model with mixed language and built the heterogeneous character dictionary tree according to the corresponding edit rules to process the query words.For the long Chinese query,we put forward spell correction model of two-way parallel.For paralle processing,we put forward the concept of reverse character dictionary tree and reverse language model.The training corpus used in the model is extracted from the user query log,click log,web links and other information.Experiment shows that the parallel query correction method for mixed language increases the accuracy by 9%,reduces the recall by 3%,and,especially,speeds up the processing by 40% compared to single pass query correction.