摘要
针对文本相似性度量中的相似度阈值、准确率、召回率、shingle滑动窗口大小、shingle权重系数和文本属性等参数相互影响、关系复杂的问题,研究了这些参数之间的相关性,并结合实际应用需求,提出各参数可优化配置的建议,分析与设计了相似度阈值可适应文本篇幅属性的相似性度量算法.通过某基金2009年的7378个项目申请书的比对分析,结果表明:提出的算法不但适用于大规模的文本集合,而且在短小的文本集合中进行相似性度量也具有很高的应用价值,其准确率和召回率均可高达95%以上.
Parameters in text similarity measurement such as similarity threshold,precision,recall rate,size of shingle moving window,shingle weighted coefficient and text attributes are interrelated and their relationship are complicated.Based on the analysis of parameter correlations and practical requirements,we suggest optimized parameter configurations and design a similarity measurement algorithm to adjust the similarity threshold to the text length contribute.The algorithm is applied to the text similarity analysis of 7378 proposals for some fund in 2009.The results demonstrate that,no matter the text length is long or short,the algorithm is so efficient that the precision and recall rate are both higher than 95%.
出处
《小型微型计算机系统》
CSCD
北大核心
2011年第5期983-988,共6页
Journal of Chinese Computer Systems
基金
国家自然科学基金项目(60873081
60970095
M0921005)资助
湖南省自然(07JJ6122)资助
关键词
文本相似性度量
算法
邻接词组
参数相关性分析
召回率
text similarity measurement
algorithm
shingle
parameters correlation analysis
recall rate
