基于主题模型与信息熵的中文文档自动摘要技术研究

Automatic Text Summarization Research Based on Topic Model and Information Entropy

提出了一种基于LDA模型以及信息熵的文档自动摘要技术,即通过LDA模型对文档进行浅层语义分析,得到文档的主题分布以及不同主题下的词语分布;通过对主题的分析,可以得到最能代表文档中心思想的主题,以及该主题下的词语分布。同时,提出了一种新的基于信息熵的度量句子重要性的方法,并将该方法应用于文档的关键句抽取过程中。该方法将文档中句子的出现看成一个随机变量,通过对随机变量建模并度量它的信息熵来选取文档中的关键性语句。实验结果表明,应用主题模型与信息熵摘取的文档摘要能有效地从文档中摘出中心句。

This paper presented a method for automatic summarization based on LDA model and information entropy for Chinese document.It uses LDA model to do shallow semantic analysis work on documents and gets the distribution of topics under each document.Through analyzing the topics of document,we got the topic which has the best expression of central idea for document.Meanwhile,this paper proposed a new method to compute the sentence weight and extract the most important sentence based on measuring the information entropy for each sentence.It treats the sentence as a random variable and calculates the information entropy for every random variable.Experimental results show that this method can pick out the most important sentence in the document.