面向词权重的主题识别应用研究

Research of Topic Recognition Based on Term Weighting

[目的/意义]在文本主题求解时,LDA模型更倾向于高频率的词项,造成主题的语义特征和内容区分度不高.[方法/过程]从文本的词权重入手,综合考虑词项在文本集合中的全局统计特征和局部语义特征,衡量词语在文本中的重要性,并将词语的特征值作为LDA主题模型的输入,改变LDA模型生成词的概率.[结果/结论]实验表明,结合词权重的LDA模型,具有更好的模型拟合度,同时能够较好的识别语料库中主要话题,提高了主题词分布的广度和主题的语义区分度.通过新闻文本数据验证了该方法的可行性与有效性.[局限]对词语的局部语义特征描述需要大数据量的计算.

[Purpose/significance]In order to recognize the text topics,LDA model tends to use high-frequency terms,which results in low semantics and content discrimination.[Method/process]Starting from the term weighting,the paper measures importance of words in text,considering both the overall statistical features and local semantic features of words in text set.Then the word feature values are used as input of LDA topic model to change the probability of LDA model generating words.[Result/conclusion]Experiments show that the LDA model combined with term weighting has better model fitting degree,and can identify the main topics in the corpus,and improve the breadth of the distribution of topic words and the semantics of topic.The feasibility and validity of this method are verified by news text data.[Limitations]A large amount of data is needed to describe the local semantic features of words.