网络舆情中的信息预处理与自动摘要算法

Web Data Preprocessing and Automatic Abstract for the Public Opinion

提出了一套从互联网原始数据中提取有效信息,利用网页模版对这些信息进行格式化整理,并依据网络舆情的特点对正文信息进行自动摘要处理,在自动摘要处理过程中引入倒排索引概念简化了算法复杂度,基于关键词的权重分析保证了算法能够更准确的找到人们关心的信息.测试结果支持了本文提出的算法.

This paper firstly introduced an approach of the data preprocessing and formatting from the original web pages based on web wrappers. Then, an algorithm of automatic abstract for the public opinion is presented. During the process of the automatic abstract operation, we use the technique of Inverted-Index to furthermore decrease the algorithm＇s complexity. And the policy of weight analysis based on key words is used to ensure people to find the most cared information. Finally, the experiment results show that the proposed algorithm is very efficient.