一种基于类别先验信息的问题检索语言模型

A Language Model Based on Category Prior for Question Retrieval

社区问答系统已经积累了大量的以层次类别结构进行组织的问题答案对。为了能够重用这些非常宝贵的历史问题答案对资源,设计出一个非常有效的问题检索模型至关重要。在该文中,我们在语言模型建模的框架下提出了一种新的基于问题类别先验信息的方法来提高相似问题检索的性能。特别地,我们将叶子类别语言模型看作是Dirichlet超参来对一元语言模型的参数进行加权,从而提出了一种新的基于类别先验信息的语言模型。该方法具有严格的数学推导依据。在来源于Yahoo!Answers的真实的大量数据集上做了实验比较和分析,实验结果表明我们提出的方法比之前简单的线性插值的方法具有非常显著的性能提升。

Community Question Answering （CQA） services have been building up large archives of question-answer pairs, which are organized into a hierarchy of categories. To reuse the invaluable historical question-answer pairs, it is essential to develop effective Question Retrieval （QR） models. In this paper, we propose a novel approach based on category prior of questions within the language modeling framework for improving the QR performance. Specific- ally, a new Language Model based on category prior is proposed which views the Leaf Category Language Model as the Dirichlet hyper-parameter that weights the parameters of the unigram Language Model. The approach has solid mathematic foundation. Experiments conducted on a large scale real world CQA dataset from Yahoo! Answers show that our proposed method can significantly outperform the previous work which just combines the category informa- tion with the unigram Language Model linearly.