基于交叉熵与困惑度的LDA-SVM主题研究

Research on LDA-SVM subject based on cross entropy and perplexity

目前对于中文影视剧本的分类主要借助人工经验,具有成本高、效率低等特点.当前没有针对中文影视剧本主题自动分类的相关研究,本文将对主题提取进行研究,传统主题生成模型借助于文档和段落、段落和语句、语句和词的相似性,而忽略了文本语句与语句之间的相似性.首先,采用ISOMAP方法降低样本集的向量空间维度;其次,提出交叉熵结合困惑度的算法模型,进而确定LDA需要提取的最优主题数目;最后,通过剧本-主题的方式,利用LDA算法挖掘剧本的隐含主题词,同时利用SVM对主题词做出进一步的分类.

At present,the classification of Chinese film and television scripts mainly relies on manual experience,which has the characteristics of high cost and low efficiency.There is currently no research on the automatic classification of Chinese film and television scripts.This paper explores the topic extraction.The traditional topic generation model relies on the similarity of documents and paragraphs,paragraphs and sentences,sentences and words,while ignoring the similarity between text statements and statements.Firstly,the ISOMAP method is used to reduce the vector space dimension of the sample set.Secondly,the algorithm model of cross entropy combined with perplexity is proposed to determine the optimal number of topics that LDA needs to extract.Based on the above,through the script-theme method,the script is used to mine implicit subject terms of the script,while using SVM to further classify the subject words.