一种基于逐层扫描的频繁字串快速提取算法

An Algorithm of Fast Frequent String Extracting Based on Level-wise Scan

串频统计是一种简便有效的抽取未登录词方法。本文提出了一种快速的频繁字串提取和计频方法,通过逐层扫描快速发现频繁字串,修正字串有效出现频次,最后抽取平均互信息量达到阚值的字串。实验结果显示该方法有效可行。

String frequency statistics is a simple and effective method of extraction unlisted word. This paper presents an effective algorithm of extracting frequent strings. It uses a level-wise scan for finding rapidly frequent strings and modifies the valid frequency that string appears in text. Finally, those high-frequent strings that reach the threshold of average mutual information are extracted. Experimental results show that the method is effective and feasible.