信息检索中一种基于词语—主题词相关度的语言模型

A Term-Subject-Association-Based Language Model for Information Retrieval

本文提出一种基于词语-主题词相关关系的语言模型TSA-LM(Term-Subject Association Based Language Model),它的基本思想是把一篇文档分成两个文档块,一部分是由领域主题词表中的主题词构成的主题词文档块,另一部分是由非主题词构成的非主题词文档块,分别计算两个文档块和查询的似然程度。对非主题词文档块,假设词语间独立无关,沿用经典的语言模型计算;对主题词文档块,把查询词语和主题词相关关系引入语言模型中来估计该文档块和查询的似然程度。词语-主题词相关关系采用词语-主题词相关度来衡量。词语-主题词相关度的计算除了来源于对文档中词语-主题词共现性的观察外,还来源于宏观上对词语-文档-主题词归属关系的观察。公开数据集上的检索实验结果表明,基于词语-主题词相关关系的语言模型可以有效提高检索效果。

We propose a Term-Subject-Association-based Language Model （TSA-LM） for document retrieval. Its main idea is to divide a document into two parts： one is only composed of subject words （named as subject block）, and the other contains no subject words （named as non-subject block）. Query-likelihood of a document is measured by the combination of the query-likelihood of the two blocks. For non-subject block, we follow classical language model. For subject block, we use the language model smoothed by term-subject association. The term-subject association is weighted by term-subject co-occurrence and term-document-subject labeling relationship. The experimental results on public dataset show that TSA-LM improves search effectiveness.