基于过滤与权重平滑策略的自动标引方法研究

Research on the Automatic Indexing Method Based on the Filtering and Weight Smoothing Strategies

文章提出一种基于过滤和权重平滑策略的标引词自动抽取方法,该方法采用可变停用词作为文档分隔手段,采用词性、词频、词语位置等信息作为标引词过滤的手段,采用合理的权重倾向策略保证了标引词抽取在词组和单词中的均衡。方法综合利用了词性标注等自然语言处理领域的研究成果和统计学信息,不依赖词汇在文档集中的分布规律,可以直接从单篇文档抽取标引词,在待标引文档篇幅受限的情况下具有良好的运行性能。

This paper proposes an automatic indexing word extraction method based on the filtering and weight smoothing strat- egies. Taking the variable disabled words as the document separation means, and the information such as part of speech, term fre- quency and words＇ position as the indexing word filtering means, the paper uses the reasonable weight tendency strategies to guar- antee the balance of indexing word extraction in phrase and single word. The method makes a comprehensive use of the research re- sults and statistics information in the natural language processing field such as part-of-speech marking, is able to extract the inde- xing words from the single document directly without relying on the distribution law of the words in the document set, and has excel- lent operating performance when the length of the document to be indexed is limited.