基于无监督学习算法的推特文本规范化

Twitter text normalization based on unsupervised learning algorithm

推特文本中包含着大量的非标准词,这些非标准词是由人们有意或无意而创造的。对很多自然语言处理的任务而言,预先对推特文本进行规范化处理是很有必要的。针对已有的规范化系统性能较差的问题,提出一种创新的无监督文本规范化系统。首先,使用构造的标准词典来判断当前的推特是否需要标准化。然后,对推特中的非标准词会根据其特征来考虑进行一对一还是一对多规范化;对于需要一对多的非标准词,通过前向和后向搜索算法,计算出所有可能的多词组合。其次,对于多词组合中的非规范化词,基于二部图随机游走和误拼检查,来产生合适的候选。最后,使用基于上下文的语言模型来得到最合适的标准词。所提算法在数据集上获得86.4%的F值,超过当前最好的基于图的随机游走算法10个百分点。

Twitter messages contain a large number of nonstandard tokens, created unintentionally or intentionally by people. It is crucial to normalize the nonstandard tokens for various natural language processing applications. In terms of the existing normalization systems which perform poorly, a novel unsupervised normalization system was proposed. First, a standard dictionary was used to determine whether a tweet needs to be normalized or not. Second, a nonstandard token was considered to take 1-to-1 or 1-to-N recovering based on its characteristics. For 1-to-N recovering, the nonstandard token would be divided into multiple possible words using forward and backward search. Third, some normalization candidates were generated for nonstandard tokens among multiple possible words by integrating random walk and spelling checker. Finally, the best normalized twitter could be obtained by taking all the candidates into consideration of n-gram language model. The experimental results on the manual dataset show that the proposed approach obtains F-score of 86. 4%, which is 10 percentage points higher than that of current best graph-based random walk algorithm.