摘要
串频统计是一种简便有效的抽取未登录词方法。本文提出了一种快速的频繁字串提取和计频方法,通过逐层扫描快速发现频繁字串,修正字串有效出现频次,最后抽取平均互信息量达到阚值的字串。实验结果显示该方法有效可行。
String frequency statistics is a simple and effective method of extraction unlisted word. This paper presents an effective algorithm of extracting frequent strings. It uses a level-wise scan for finding rapidly frequent strings and modifies the valid frequency that string appears in text. Finally, those high-frequent strings that reach the threshold of average mutual information are extracted. Experimental results show that the method is effective and feasible.
出处
《计算机科学》
CSCD
北大核心
2008年第5期127-130,共4页
Computer Science
关键词
频繁字串
中文抽词
逐层扫描
互信息
Frequent string,Chinese automatic word extraction, Level-wise scan, Mutual information