摘要
为克服传统词频—逆向文本频率(TFIDF)关键词提取精度低下的缺点,文章提出一种基于多级统计特征的关键词提取(TFIDF-SK)算法。该算法采用词语TfDf指标的离散系数公式来剔除噪音词,构建基于词偏度、词语位置权重信息和词频—逆向文本频率的评估函数来度量关键词的重要性。实验结果表明该算法优于传统方法,在大数据时代网络情报监测中具有广泛的应用价值。
In order to overcome the shortcomings of classical TFIDF - based keywords extraction method with low accuracy, keywords extraction algorighm (TFIDF - SK) based on multi - level statistical characteristics is proposed. It firstly uses coefficient of variance formula of the word' s TfDf index to remove noise words, and secondly constructs evalu- ation function based on word's skewness, word's position weight and word frequency - inverse document frequency to measure the importance of keywords. Experimental results show that the algorithm (TFIDF - SK) outperforms the traditional method, and has a wide range of applications in the Internet intelligence monitoring in the big - data era.
出处
《情报资料工作》
CSSCI
北大核心
2013年第3期64-68,共5页
Information and Documentation Services
关键词
大数据
关键词提取
词偏度
网络情报监测
big data, keywords extraction, word's skewness, Internet intelligence monitoring