基于信息熵和词频分布变化的术语抽取研究

Term Extraction Based on Information Entropy and Word Frequency Distribution Variety

在分别研究了基于信息熵和基于词频分布变化的术语抽取方法的情况下,该文提出了一种信息熵和词频分布变化相结合的术语抽取方法。信息熵体现了术语的完整性,词频分布变化体现了术语的领域相关性。通过应用信息熵,即将信息熵结合到词频分布变化公式中进行术语抽取,且应用简单语言学规则过滤普通字符串。实验表明,在汽车领域的语料上,应用该方法抽取出1 300个术语,其正确率达到73.7%。结果表明该方法对低频术语有更好的抽取效果,同时抽取出的术语结构更完整。

A term extraction system based on information entropy and word frequency distribution variety is presen- ted. Information entropy can measure the integrality of the terms while word frequency distribution variety can measure the domain relativity of terms. Incorporating with simple linguistic rules as an addition filter,the automatic term extraction system integrates information entropy into word frequency distribution variety formula. Preliminary experiment on the corpus of automotive domain indicates that the precision is 73.7% when 1,300 terms are extrac- ted. The result shows that the proposed approach can effectively recognize the terms with lower frequency and the recognized terms are well of integrality.