信息检索用户查询语句的停用词过滤

Removal of Stop Word in Users’ Request for Information Retrieval

针对以自然语言形式提出的查询请求,区分信息需求表述和信息内容两部分。基于近20万语句的查询语料库和背景语料人民日报对照,提出汉语通用停用词和查询专用的相对停用词,采用左右熵和Ngram方法及KL距离脱机构造相应候选词表。根据候选词语的Bigram属性和句中不同位置的分布特点,给出了在线动态识别停用词的方法。实验结果表明,该文的方法比单纯根据静态停用词表标注效果要好。

Information need expression and information content words are distinguished for users requests in natural language. Based on the analysis of 200 000 query sentences and the People＇s Daily corpus, absolute stop word and relative stop word are proposed. The candidate stop word lists are built offline by means of left/right entropy, Ngram and KL divergence. With the information of Bigram and different position distributions, this paper gives a dynamic identification algorithm for the actual stop word in users＇ request expression. The experiment shows the method is superior to the baseline which only consults a stop word list.