基于分层狄利克雷过程模型的文本分割

Text Segmentation Based on Hierarchical Dirichlet Processes

文本分割在文本摘要、信息检索等诸多领域都有重要的应用。主题模型是该领域研究中的重要方法,但目前基于主题模型的方法普遍依赖于主题个数的人工设置。针对此问题,本文提出了一种基于分层狄利克雷过程(Hierarchical Dirichlet process,HDP)模型的文本分割方法。首先使用HDP模型获取文本在主题空间的向量表示,然后将主题向量用于C99分割算法实现文本分割,最后使用两种优化策略对结果进行优化。实验结果表明,基于HDP模型的方法能够摆脱对人工设置主题个数的依赖,有效提高了文本分割的性能。

Text segmentation has important applications in many fields,including text summarization,information retrieval,and so on.Topic model is an important tool in text segmentation.However previous text segmentation methods based on topic model generally rely on manually setting of the number of topics influencing results significantly.To solve the problem,a novel text segmentation method based on hierarchical Dirichlet process（HDP）model is proposed.Firstly,texts are modeled with HDP model to get their expression with topic vectors.Then,the topic vectors are used in C99 segmentation algorithm for text segmentation.Finally,two optimization strategies are applied to result optimization.Experimental results show that the presented method can omit manually setting of the topics numbers and improve the performance of text segmentation.