大数据时代基于统计特征的情报关键词提取方法

On the Statistical Features-based Information Keyword Extraction Method in the Era of Big Data

为克服传统词频—逆向文本频率(TFIDF)关键词提取精度低下的缺点,文章提出一种基于多级统计特征的关键词提取(TFIDF-SK)算法。该算法采用词语TfDf指标的离散系数公式来剔除噪音词,构建基于词偏度、词语位置权重信息和词频—逆向文本频率的评估函数来度量关键词的重要性。实验结果表明该算法优于传统方法,在大数据时代网络情报监测中具有广泛的应用价值。

In order to overcome the shortcomings of classical TFIDF - based keywords extraction method with low accuracy, keywords extraction algorighm （TFIDF - SK） based on multi - level statistical characteristics is proposed. It firstly uses coefficient of variance formula of the word＇ s TfDf index to remove noise words, and secondly constructs evalu- ation function based on word＇s skewness, word＇s position weight and word frequency - inverse document frequency to measure the importance of keywords. Experimental results show that the algorithm （TFIDF - SK） outperforms the traditional method, and has a wide range of applications in the Internet intelligence monitoring in the big - data era.